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Two reviews of NIE's study of compensatory education 
are provided in this document. The conclusions of the first reviewer 
(House) are that: (1) the study is valid because it meets the 
standards for the evaluation of a public program; and IZ) trie 
compensatory programs, especially Title I, are very good. Other 
conclusions are that the funds' allocation requirements are necessary 
but the program development requirements are excessively regulated, 
program management is efficient but monitoring is confused, and how 
much students develop in these programs remains hasy. The second 
review, "Evaluating Compensatory Education" (Husen), takes a broader 
oersoective, based in part on the reviewer's earlier participation in 
the Organisation for Economic Cooperation and Development's review of 
U.S. educational policies. Against this background, the reviewer 
discusses the role of education in the United States, the philosophy 
and strategies behind compensatory education. Title I targeting, and 
evaluation problems. This review concludes that? (1) in looking at 
the outcome of intervention programs,, a broader perspective is needed 
than appeared in some of the HIE evaluation efforts; (2) **^Pl ann i cd 
small-scale longitudinal studies conducted where different models of 
compensatory education are operated would prove more than large-scale 
surveys; and (3) compensatory education programs are only small 
changes in the larger contest of formal education, end minor chances 
in input should not be expected to produce dramatic results. I CMC) 
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£rn«st R. House 
University of Illinois** 

Compensatory education has been the largest ^ 
and most controversial area of federal involve-' 
ment in education since its beginning nearly two 
decades ago. The evaluation of compensatory ed- 
ucation has been no less controversial. Bpth 
touch deep-seated value and interest conflicts 

• ^ i 
in society. 

\ 

Tn 197«4, the Congress instructed the Na- 
tional Institute of Education (NIE) to conduct a 
study of compensatory education, especially 
those programs funded by Title I of the Elemen- 
tary and Secondary Education Act, which has pro- 
vided over $2 billion annually for educational 
programs for low-achieving students in school 
districts with low-income families. The NIE 
mandate grew out of Congress's frustration with 
past evaluation efforts. ' 



**Ern'_-st House wishes to acknowledge the 
helpful comments provided by Gene Glass, Paul 
Hill, Douglas Weir, Gordon Hoke, and staff from 
the Illinois Board of Education — Tom Kerins, 
Connie Wise, Rose Maye , and Jack Fyans. 
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By 1974, the evaluation of compensatory ed- 
ucation in general and Title I in particular had 
indee.d been inadequate. After several years of 
federal programs, and several hundred millions „ 
of dollars spent on the evaluations, Congress 
« had little information on which to bas^ deci- 
sions- As a member of the staff of the House 
Education and labor Committee put it: "Incred- 
ibly enough, both the sponsors and opponents of 
that provision found that almost no data existed 
on how much money from. the Title I allocation 
was then being spent on basic skills" (Cross, 
1979, p. 16). 

TK© ^reasons for this inadequacy are found 
in the history of the Title I evaluation and in 
the type of evaluation that has been attempted 
for the past decade- Put simply, federal offi- 
cials in the Office of Education have attempted 
to evaluate these programs by an inappropriate 
approach to evaluation and when the approach 
fails, they modify the evaluation slightly and 
attempt the same approach again, and it fails 
again , This has occurred repeatedly over the 
past fifteen years (McLaughlin, 1975). The re- 
suit has been either no information at all or 
evaluation data that are severely criticized as 
invalid by evaluation experts. Controversy suc- 
ceeds controversy, 
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frustrated by* this problem as early as * 
197** f Congress tried two solutions. One was to 
write into the Title I legislation itself a re- 
quirement that the Office of Education develop 
"models of evaluation" and that local school 
districts evaluate their programs using these 
models* As in many cases where Congress tries 
to legislate the detailed conduct of technical 
activities, the legislation led to questionable 
results* 

The Office of Education developed three 
models of evaluation which emulated the unsuc- 
cessful approach to evaluation that thte Office 
had been using* Although the legislation was 
Enacted in 1074, there is still little usable 
infdfrmation on which to base decisions. In ad- 
dition, the evaluation models themselves are 
under attack by evaluation and" measurement ex- 
perts, as well ar. by the state evaluators whom 
the models are supposed to help (Linn, 1979; 
Jaeger, 1979; Wiley, 1979). The state evalua- 
tion directors have strongly protested the im- 
position of these particular models, yet the 
work goes forward. In 1979, use of the three 
models was mandated for all Title I programs. 

9 

The other Congressional solution to the 
problem was to instruct the f National Institute 
of Education, an agency 'separate from the Office 
of Education, to conduct a study of compensatory 
education, especially the Title I programs. It 
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is my happy duty to report that the federal gov- 
ernment has redeemed its bpnov ( and its compe^ 
tence in this study. It has conducted an evalu-^ 
ation that was helpful to the Congress, that was 
valid, that was worthy of the name. 

The former staff member of the House Educa- 
tion and Labor Committee commented: 

More "recently the attention of Congress 
with respect to the reenactment of Title I 
has been focused to a remarkably large 
degree by information generated by the 
stud,y done by the National Institute of 
Education. . .The NIE study has been 
remarkable in terms of the quality and 
quantity of information that it has pro- 
duced . . . One of the more specific 
products to' emerge from that study was 
substantial revision and reorganization of 
the Title I statute. ♦ .At this ^point one 
could say that the NIE study has been the 
single most productive piece of evaluation 
work ever undertaken on a federal education 
program. It may well be one of the most 
productive evaluation works of any 
executive branch agency* (Cross, 1979, pp. 
20-25.) 

This burst of enthusiasm is understandable 
after so many years of failure. The intention 
of this review is to analyze the NIE study, to 
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judge its worth, and to assess its , strengths and 
weaknesses. 'I shall try show why it is a 
good study and to outline the critical decisions 
made by the evaluators and officials that -kept 
it from going astray. I suggest in some places 
how it might have been made even better than it 
was. 

♦ 

Judging Evaluations 

•» 

What is Tt that makes an evaluation good or 

Hi 

bad, valid or invalid? Suppose the evaluator 
and the decision maker are the same person, as 
when a person is deciding to purchase a car. 
What is the minimum standard for the evaluation? 
In this case one would say that the evaluation 
must be true. If the evaluation were not true, 
even though conducted by an individual for him- 
self or herself, one would say the evaluation 
was invalid. There are many different ways in 
which individuals determine truth, and, indeed, 
these different methods distinguish evaluation 
approaches. But, whatever the method, an untrue 
evaluation would be considered invalid by every- 
one'. It would make little sense to say, "This 
is a valid evaluation, but it- happens to be un- 
true." (Valid/ty I take in the sense of being 
worthy of recognition.) 

-^Now consider the case when the evaluator 
and decision maker are separate persons. Per- 
haps the decision maker consults Consumer Re- 
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ports to make a decision. It is not" enough that 
the evaluation be true; it must al30 be credible 
to the decision maker. An incredible evaluation 
is likewise invalid, i..e., not worthy of 
recognition. Again, thera are different ways in 
which credibility is assessed. Agai.., it would 
not make much sense to say, "This is a valid 
evaluation but it is not credible." 

Finally, consider a third "case. In .this 
tease, the evaluator is not evaluating something 
for an individual's private use, but is evalua- 
ting' a public program, a social program. - In 
this, case the evaluation must not only be true 
and credible, it must be normatively correct. 
This is the most stringent demend of all, for 
one has passed from the private into the public 
realm, and there are obligations to the public 
at large and not simply to private individuals. 
This is the area of least agreement. I will not 
make an extended argument here but will simply 
contend that an evaluatidh of a public program 
must be democratic, , fair, and. impartial. If it 
is undemocratic or unfair, it is invalid. It 
makes little sense tp say of an evaluation of a 
public program that it is val*Sd but that it is 
un,just, unfair, or partial. (For development of 
these ideas, see House, 1980. ) 

Why this 'is so is ^ecause of the role of 
evaluation in our social decision procedures, in 
determining who gets what. To be valid a pri- 
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vate evaluation must be true, an interpersonal 
evaluation must be true and credible, and a pub- 
lic evaluation, must be true, credible, ■ and 
right, i.e., democratic, fair, impartial, and sc- 
on. The burden of those who presume to evaluate 
social programs in the public interest is not a 
light one — nor should it be. It seems only 
fair and just that evaluators should meet stan- 
dards that they expect of others. * -I shall at- 
tempt to show in this review that the NIE study 
meets these standards of validity. 

In fact, I would contend that the ' success, 
acceptance, and utility of the study is signif- 
icantly dependent on- its meeting these validity 
claims. In the words of a Congressional staf- 
fer: * 

i 

• m # Congress is generally not concerned 
withKthe technicalities in the development 
of evaluations. What the Congress want3 is 
information on which ta base policy ana 
political judgments about programs. The 
primary concern of Congress is that the 
information /that is produced can withstand 

w i 

a professional critique and that the data 
will produce findings that are at least 
somewhat conclusive. (Cross, 1979, p. 19.) 

^ n 

Although part of the judgment as to whether 
the evaluation is good depends on whether it can 
be used by its intended audience (in this case, 
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the Congress), utility of. the J.n for nation alone 
j.s not a sufficient criterion tc determine the 
validity of the evaluatidn. Congress 'may make 
good use of information which »io false. Or an 
evaluation that is good for Congress may not be 
good for the country as a *hole. The Congres- 
sional interest may not be identical with- the 
public interest. An evaluation* could be good 
for Congress's own -purposes and iatsreJt but 
invalid in that it did not serve the public in- 
terest. Thus utility for an intended audience., 
while important, is an 1 insufficient criterion 
when evaluating public programs, evenjwhen the 
audience is the government- 

From the quote above, it would appear that 
Congress wants relevant information and albo in- 
formation the truth and credibility of which are 
assured by professional consensus. Another cri- 
terion — that of conclusiveness — I will not 
"address other than to say that government -offi- 
cials often expect a conclusiveness that is be- 
yond the power of formal evaluation or social 
science to deliver. (For a discussion, see 

r. 

Lindblom and Cohen, 1979). Congress is actually 
looking for practical information that can be 
used to' impVove programs.. The NIE study, while 
inconclusive in many ways, was able to provide 
such practical information and hence ' satisfied 
the Congress. - *tore elusive* are enduring" gener- 
alizations abouf Social programs and social 
phenomena. Pursuit of, these has led to incon- 

-365- . , 

10 



elusive evaluations and to Congressional frus- 
t rat ions. 

In summary, t\m conclusion of my review is 

that the NIE compensatory education study is 

valid because" lit meets" the standards for the 

evaluation of a public prpgram, and it is a good 

# • # 

evaluation because it was used by and satisfied 

its intended audienqe, the Congress. Its value 
to Coifgress was partially dependent on its va- 
lidity/ It would have been a valid evaluation 
even if, thrdugh sAne strange circumstance, Con- 
gress had never seen it or used it, , in which 
case it' would have been of* no value to them. 

The Charge \ 

. ' Section 821 of the Education <Am$ndxnents of 
197U instructed the NIE to undertake a "thorough 
evaluation and study*^of compensatory education 
programs* Such study had to include the follow* 
mg: 

1. An examination of purposes and effec* 
tiveness of these programs; 

2. An* analysis of means to identify 
children who have^ the greatest need 
for such programr; 

>3. An analysis of the effectiveness of 
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methods and procedures for meeting the 
educational needs of children, includ- 
• i'ng the use of individualized plans 
and teacher training programs; 

4. An exploration of alternative methods 
of distributing funds in an equitable 
and efficient manner that ensures the 
funds reach the areas of greatest 
need; 

5. Establishment of not more than 20 ex- 
perimental programs to carry out the 

above purposes. * 
¥' • » 

In addition to allowing four years for com-^ 
pletion, the law instructed NIE not to report 
the- results to any other group, e.g. the Presi- 
dent or the Commissioner of Education, before it 
reported to the " Congress. Specific Congres- 
sional concerns in' passage of the act were the 
effects of altering the definition o* poverty, 
of using t^st scores to distribute funds, of re- 
quiring individualized plans for all students in 
Title I programs, #nd of requiring" school dis- 
^icts to spend 85% of their Title T grants oh 
basic skills instruction. 

From these concerens NIE designed a . wide 
focus study, one purpose of which was to assess 
the past effectiveness of the program and the 
other to examine specific alternative proce- 
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dures* So the study was both an evaluation ana 
a policy study. The wide foe* . was the most 
critical decision — and a wise one* Past eval- 
uations had foundered by using standardized 
achievement scores as the exclusive indicators 
• of program jout comes. In both correlational and 
experimental designs, the Title I programs re- 
peats 1 to show large increase* on stan- 
c rdized te; Measurement and evaluation ex- 
pe *ts critic: ^ed the narrow focus of the out- 
come ^ , the experimental designs, the instruments 
employ d, and the statistical techniques, thus 
throwing the evaluations into question. 

By contrast the NT^ officials concluded 
,that compensatory e ducat had a number of im- 
portant goals, most of which were not captured 
by outcomes on standardized achievement tests, 
which were themselves considered by many * to be 
invalid measures. Although there were many mo- 
tives for originating compensatory education 
programs, such as eliminating poverty*, the spe- 
cific purposes clearly discernible in the status 
and Congressional statements were to provide fi- 
nancial assistance to districts with low income 
children , to fdnd special services f ©r low- 
achieving children in the poorest schools, and 
to contribute to the overall development of stu- 
dents. Members of Congress themselves differed 
on the priority of the three fundamental pur- 
poses. There was no reason to assume (as past 
evaluators had done) that the programs existed 
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solely to enhance academic achievement as meas- 
ured by tests. 

Furthermore, federal involvement operated 
through state and local agencies which had their 
own goals. The evaluation had to consider im- 
plementation as a factor. This insight into 
compensatory education programs, particularly 
Title I programs, as having multiple arid pos- 
sibly conflicting goals and as being implemented 
by other agencies with their own structure and 
purposes, is what distinguishes the success of 
this evaluation from past failures. It is es- 
sentially a political insight, and it is sig- 
nificant that the director of the study (Paul 
Hill) 1 had a background in political science 
rather than economics, engineering, accounting, 
psychology, or even sociology. The temptation 
in all these disciplines would be to eecure 
overly reductionist outcome measures. (Of 
course, political science is also reductionist 
in its focus on power and authority.) 

The NIE strategy was designed to produce a 
"complete understanding" of how Title I operated 
by focusing on funds allocation, service de- 
livery, student development, and program ad- 

* ministration, and by .employing criteria derived 
from the fundamental purposes. The study did 
not provide a complete understanding, but it did 

'provide information that Congress could and 
would use to judge and improve Title I. .The in- 
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itial definition of the problem derived from the 
Congressional charge was critical to the success 
of the study. Other, less fortunate, interpre- 
tations were possible. Although most evaluation 
discussions focus on technicalities of design 
and statistics, it is in the initial conceptual- 
izing of the problem that the essential features 
are determined, usually beyond remedy. 

Needs 

A reading of the original Congressional 
mandate reveals how easily the NIE study could 
have gone astray* The mandate is implicitly 
written from the "needs assessment" paradigm: 
determine the purpose of the programs, analyze 
means of identifying children in need of such 
programs, and analyze the effectiveness of meth- 
ods for meeting the needs* Needs is a complex 
and unquestioned concept, and when used as in 
"children's needs," a clear moral mandate. The 
existence of a need is used to justify acting in 
a certain fashion* Most often, need is used to 
indicate the discrepancy if between some target 
state and the actual state of affairs. (See 
Scriven and Roth, 1977, and Roth, 1977, for a 
half-dozen different ways in which needs are de- 
fined. ) 

It is clear enough that children need vi- 
tamin C or so many calories a day, but the fur- 
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ther one moves away from biology, the less cer- 
tain the concept of need becomes. For example, 
the child needs a strong self-concept (mental 
health) , but people disagree immediately as to 
what this entails. Typically, needs assessments 
in compensatory education programs are conceived 
as the need to reach the average level of educa- 
tional achievement, as operationally defined by 
a standardized achievement test. 

Such a construal of needs could have re- 
sulted in another evaluation such as occurred in 
the past. The study might have been designed as 
a larRe-scale field experiment to see whether 
Title I programs improved children's test scores 
or brought them up to grade level on standard- 
ized tests. An experimental design would be 
necessary for determining whether the expected 
gain was attributable to the programs and not to 
other causes. Yet there are many reasons why 
children do not perform at grade level on tests, 
including family background, past history o*f 
•disadvantagement, and even ■ the nature of the 
tests themselves. This type of evaluation makes 
the federal program solely responsible for ele- 
vating test scores when there are many causes of 
low test scores. The programs are likely to 
look like failure*.. They are, after all, only 
supplements to the children's education. 

Thus the net effect might be to make the 
programs and those who run them appear incompe- 
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tent. President Nixon justified no increase in 
funding for compensatory programs because he 
said educators did not know what worked, a di- 
rect reference to past evaluations * Often the 
children themselves were blamed for the failure. 
Some researchers have concluded that certain 
children are intrinsically incompetent because 
compensatory ^programs have f not succeeded, ac- 
cording to this type of evaluation. 

In this case the Congressional mandate in- 
structed the researchers to examine the purposes 
of the programs and to analyze the means for 
identifying the children who have the greatest 
need for the program. The researchers defined 
the set of fundamental purposes as providing fi- 
nancial assistance, funding special services for 
low-achieving students in poor schools, and con- 
tributing to the cognitive, emotional, social, 
or physical development of these students. Needs 
were not reduced to test score performance, al- 
though there was a testing component to the 
study. The financial assistance to poor stu- 
dents and the provision of special services were 
conceived as positive outcomes. In other words, 
the needs were for more money for the education 
of poor students and for special funding for 
♦them,, as well as a need to enhance their overall 
development. 

Other possible purposes for the programs 
would have been to eliminate poverty and to re- 

-372- 



- 17 



9 

ERIC 



distribute wealth. An egalitarian (and I am 
one) might hope for the programs to reduce pov- 
erty among the students. But, again, fsince pov- 
erty has many causes, many of which are external 
to the school's influence, expecting the schools 
to alleviate poverty puts an extraordinarily 
heavy burden on the programs and one the pro- 
grams probably cannot carry. It would be inter- 
esting to know the degree to which the programs 
alleviated poverty, although how this might have 
been determined in a four-year evaluation of 
public school children is not at all clear. 
What the study did do is examine the immediate 
redistributive effects of the programs. 

One of the internal conflicts in compensa- 
tory programs is that they are supposed to com- 
pensate for both poverty and educational dis- 
advantagement . These two goals do not always 
coincide, and many decisions in the study re- 
flect a weighing of the two, and a trade-off of 
one against the other. This balancing of cri- 
teria applies to the evaluation of any complex 
social, program. 

In summary, what one takes as the purposes 
of the programs will very much influence what 
one takes as the children's needs and will alter 
the shape and results of the evaluation. Child- 
ren's needs could be construed variously as the 
need not to be poor, to receive extra education- 
al help, or to perform at grade level on a test. 
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(There are hidden, and perhaps implausible, pre- 
sumptions that the higher test score will lead 
to better things in life later on.) In this 
study I believe that the researchers have de- 
fined the probTeTtt^a such a way that the results 
were useful. 

Funds Allocation 

One of the three fundamental purposes of 
^compensatory education, as defined by the NIE 
Study, was to provide financial assistance to 
schools in relation to their numbers of low-in- 
come children. The NIE e valuators conducted an 
in-depth analysis of the distribution of the 
Title I funds. Did the funds go to districts 
and schools with poor children? The persistent 
pursuit of this question, I think, makes the NIE 
study democratic. - 

Democracy means that everyone's interests 
are considered, and in some cases justice is de- 
fined as attention to the interests of those who 
are most disadvantaged. In this case the pri- 
mary interests are those of the children who* are 
poor. Whether they, in fact, received the funds 
that were directed to them was a critical ques- 
tion. If it is an obvious one, it should be 
noted that up to the time of the NIE study, lit- 
tle information existed as to the destinations 
of the funds. One may also wonder whether the 

-374- 



9 

ERLC 



funds do any good. First, though, they must ar- 
rive . * 

m 

A just compensatory education program, 
justice being democratically conceived, will de- 
liver the funds to the disadvantaged effective- 
ly, and a just evaluation will investigate 
whether the program does so. Poverty programs 
that help the upper classes more than the lower 
are all too common. Perhaps the greatest 
strength of the NIE study was its concern about 
these issues. 

Who gets the money? This question is not 
as easily answered as one might think. Accord- 
ing to the NIE study, the funding pattern of 
Title I is complex. By law, school districts 
are supposed to receive HQ percent of their 
average expense for each child between the ages 
of. 5 and 17 identified as eligible. 2 In fact, 
districts receive only about 17 percent. Feder- 
al rhetoric is stronger than federal action. 

• • 

The complexities of governmental jurisdic- 
tions complicate the allocation process. There 
are three steps. First the federal government 
distributes monies based on county census data. 
The states in turn distribute to school dis- 
tricts within counties, and the separate dis- 
tricts distribute to individual school within 
their jurisdictions. Eligible children are 
identified as those below the poverty level, 
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those from families receiving dependent children 
aid, and those supported by public funds in fos- 
ter homes and institutions. Poverty level is. 
• defined by the Orshans^y index, which estimates 
the 'cost of food for an adequate diet, then mul- 
tiplies by three, one-third being the proportion 
of food costs in the average budget. 

The distribution effects of the Title I 
funding formula were that counties with the most 
eligible children were indeed receiving the most 
money. There is a clause in the formula * that 
bases funding partially on average costs of edu- 
cating so that central cities and suburbs in the 
Northeast were receiving the most dollars per 
eligible child while rural Southern counties re- 
ceived fewest dollars per child. This adjust- 
ment seems reasonable since educational costs do 

i, 

differ, and there- is no cost of living adjust- 
raent built into the poverty definitions. 

» 

The redistribution effects of Title I can 
be seen by the fact that when all school child- 
ren are included (as opposed to those eligible) 
the central cities receive §43 per child, the 
suburbs $20 per child, and the nonmetropolitan 
areas $40 per child. In other words, the sub- 
urbs receive less because i they have fewer eli- 
gible children. Similarly, the Southeast region 
is the most favored region of the country, 
according to the study. As to race, although 
-most elibible children are white, only 15 per- 
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cent of the white children^ are from low- income 
families, compared to H2 percent of black child- 
ren and 2 8 percent of Spanish-surnaraed children. 
Overall, half the funds that would have gone to 
suburbs on the basis of total school population 
go to central cities and nonmetropolitan areas. 
Central cities also gain 10 percent more from 
the cost component than they would by strict 
child count. There is a limit put on the cost • 
factor that stipulates that no state may receive 
'less than 80 percent or more than 120 percent of 
the national average per pupil expenditures. 
This* cost limitation favors the South and pen- 
alizes the Northeast. 

The other' factor that influences the amount 
of funding is the number of children in families 
receiving state aid and the number in institu- 
tions and foster homes. Most of these children 
are in the five large industrial states. This 
factor results in a transfer from nonmetropoli- 
tan areas to cities. The South is^the big re- 
gional loser and the Northeast the big gainer. 
Thus, different factors in the overall formula 
haVe quite different effects. 

• Allocation to counties is only the first 
step. The states must allocate funds to dis- 
tricts within counties. Most states follow pro- 
cedures dependent on the federal formula. How- 
ever, five states allocate funds entirely on the 
basis of dependent children. This results in 
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shifting an extra 10 to 16 percent of the funds 
to cities which contain high percentages of min- 
orities. These allocations are based 6n data 
that are several years old. None of the allo- 
cation procedures yield totally consistent or 
current information. 

Within a school district, funds must be al- 
located to schools in which there are high con- 
centrations of children from low-income famil- 
ies. There are a number <*>f ways by which dis- 
tricts allocate funds to schools, and districts 
are given great latitude in which procedures 
they may follow. Eligibility cah vary consider- 
ably, depending on, which indicator of low income 
is used at the school level. Districts can in- 
crease the number of eligible students by using 
different measures for 1 different schools. The 
NIE researchers seem to disapprove of this in- 
consistency, but I think it merely emphasizes 
that measures on which any funding is based are 
at best only partial indicators of what they 
supposedly represent. Obviously, consistency 
achieved by using only one measure of low income 
will be purchased by excluding some children 
identified by the other measures. In fact, I 
would encourage the use of multiple measures for 
each school. ^ 

Once schools are identified, almost all of 
them # receive services. In half wthe districts 
the amount of funding is determined in propor- 
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tion to the number of low achievers t not on the 
basis c f low income. In many districts the bas- 
is for allocation decisions is not clear. Edu- 
cational "need" is cited by some. 

to 

The idea of serving both children from low- 
income families and low achievers is built into 
the program itself. Funds awarded to a school 
because it is in a low-income area may go to 
helping low achievers in the school who are not 
from low-income families*. * Personally, I see no 
reason for limiting the services to low achiev- 
ers. It seems to me that a gifted youngster who 
is from a low- income f amity deserves extra help, 
too. So certainly should low achievers, 
whatever their income level. It would make more 
sense to target the funds to all children in 
low- income schools, since all the children in 
that school have a disadvantage, whatever their 
achievement. One could have separate programs 
for low achievers. The current system penalizes 
some poor children. There is no reason to be- 
lieve" that poverty has not stunted their life 
chances, even though they score above some arbi- 
trarily defined test' level. , 

• 

Various suggestions have been made for 
changing the definitions" of poverty. Most would 
result in significant changes in allocation. 
Census surveys tend to underestimate poverty, 
and more recent surveys have indicated that the 
South has relatively less poverty and the North- 
/ -379- 
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east relatively more than before. The groups 
studying this issue have reached no recommenda- 
tions. The measurement of poverty itself (like 
all measurement) presents complex problems. 

Title I funds are supposed to supplement 
rather than supplant t local funding. In other 
words local districts are not supposed to re- 
place their spending with federal funds. By 
comparing spending patterns in similar districts 
with and without Title I funds., the investiga- 
tors estimated . that a Titl^e I dollar increases 
local spending by 72 cents. This compares fav- 
orably with Hi cents on the dollar for other 
federal programs, 25-40 /cents for revenue- shar- 
ing funds, and 13 cents ,f or state aid. In other 
words-, Title I funding is comparatively quite 
effective in raising spending on education, 
rather* than merely providing tax relief. 

Other NIE analyses assessed the redistrib- 
utive effects, the degree to which Title I pro- 
vided more money to low-income districts. Title 
I was markedly more re distributive than" other 
state and federal programs. It allocated 5-1/2 
times as much aid per pupil • to districts with 
the highest poverty rates- (grouped by quarters) 
than to those with the lowest. The ratio for 
other federal programs was 3-1/2. Title I may 
be the most redistributive federal program.' 
Title I funding has a correlation of -0.76 with 
county per capita income. By contrast, state 
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aid programs were not very re distributive* shott- 
ing a ratio of only 1. 2-wi thin-state. 

On the other hand^. Title I does not equal- - 
ize- expenditures. Low-spending districts are 
not necessarily the low- income districts. Equal- 
ization programs provide ~ more money to low- 
spending districts. The cost factor in the Ti- 
tle I funding formula gives more money to high 
expenditure districts; Tilde I has a modest 
equalizing. ratio of 1.6 of aid per pupil of the 

.lowest to the highest quarter. A strong equal- 
izing formula would likely decrease the redis- 
tribution effects. All in all, Title I is 
strongly redistributive in effect, aiding those 
districts 0 not favored by state and local reven- 
ues. In -terms of social justice, the Title I 
programs are commendable , and the NIE study is 

.to be commended for investigating these issues. 

3 ' t 

Allocation by Testing 

A particular stimulus for the NIE study 
within Congress ys that Representative Albert 
Quie (R-Minn) was interested in using test 
scores rather tjian poverty indices to allocate 
funds. Some -people have argued that achievement 
is the "ultimate aim" of compensatory education^ 
and that test scores are a better basis for dis-" 
tribution. Others have argued that a fundamen* 
tal .purpose is to channel funds to the poor 
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rather than to low achievers. 



NIE was instructed to investigate this is- 
sue and carried out a series of substudies that 
were particularly well conceived. The investi- 
gators reasoned that such a testing system 
should be technically accurate and not suscepti- 
ble to manipulation. They recognized that any 
setting of a minimum skill level is arbitrary 
and that there are no technical procedures for 
translating standards into test items. Further- 
more, the test cut-off levels have strong ef- 
fects on the distribution, of funds. 

The investigators concluded that tests 
would have to be administered at least every 4 
years since test scores fluctuate considerably* 
To prevent various forms of cheating, the tests 
would have to be administered by trained proc- 
tors under strictly monitored conditions* No 
current testing system is adequate to distribute 
funds among states, and few state testing pro- m 
grams are adequate to allocate among districts 
withxh states* The National Assessment of Edu- 
cation Progress was designed to produce national 
rather than state-by-state estimates fluid does 
not test in all states. ¥ Current testing pro- 
grams are adequate to the task of allocating to 
schools within .most districts. 

Hence, allocating Title I funds by test 
scores would require the establishment of a new 
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testing urogram. Assuming that the testing was 
limited to. reading, to 9-year olds,, and to 10 
percent precision, a 3-year testing program 
would cost $7 million for one set of results. 
For each grade level added, costs would increase 
by 20 percent and by 50 percent for each addi- 
tional school subject. A testing program of 5 
percent precision would cost $16. & million, ac- 
cording to the NIE study. (The sampling error 
of the census data on which the poverty indices 
are based is very small.) To produce estimates 
for each school district in the country would 
cost $5 3 million. I would not argue with those 
figures other than to say they are conservative 
estimates made several years ago before the huge 
increase in travel costs. Half the total costs 
are in the data collection and would be substan- 
tially higher today. In any case, the mainten- 
ance of such a testing program would be costly. 

Assuming a cut-off at the fifteenth percen- 
tile, a figure which would yield the same number 
of students as current Title I funding proce- 
dures, NIE investigated the effects of switching 
from current allocation procedures to test score 
allocations. Twenty- three states would have in- 
creases or decreases of eligible students in ex- 
cess of 15 percent. Many of the changes would 
be quite dramatic. Generally, nonmetropolitan 
areas would lose, and central cities with high 
black populations would gain. The higher the 
cut-off score, the more the suburbs would gain. 
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If the cut-off were raised above 15 percent, the 
distribution of eligible children would resemble 
the distribution of the general child popula- 
tion. There would be no significant changes 
from region to region. 

Distribution of funds within states would 
require uniform regulations for all states, 
since state testing programs differ signifi- 
cantly, and it would necessitate major changes 
in those state programs, according to the NIE 
study. My own judgment is that the rules and 
regulations would be far more cumbersome and 
directive than expected. 

To determine what effect such an allocation 
might have within school districts, 13 ''school 
districts were given dispensation to allocate 
funds internally by test score. The overall ef- 
fect was that they served more schools and stu- 
dents than before but did so less intensively. 
They concentrated less on poor children and on 
minorities. The teaching time and strategies 
were the same. As the number of students served 
by the programs increased, students were in- 
cluded from higher income areas. On average, 
there was a 71 percent increase in the numBer of 
students and only a 44 percent increase in the 
number of teachers. Students spent le/s time in 
special instruction. The average time loss for 
each student was 14 percent,' due tt> a 13 percent 
reduction in the staff /student ratio. The study 
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of those demonstration? districts lasted two 
years. NIE investigators concluded that one 
might reasonably expect the intensity of in- 
struction to decrease still further since many 
of the activities were supported with extra 
funds not available over a longer period of 
time. 

Although NIE took no stand on using test 
scores to allocate funds, I find their studies a 
strong negative assessment of such a prospect. 
To allocate so would require the establishment 
of another large and expensive national testing 
program. It would require uniform regulations 
and substantial changes in current state testing 
programs, plus a bureaucracy to enforce such 
changes. Even then, the data would be based on 
only a thin slice" of achievement, perhaps one 
subject at one grade level. Funds would be 
shifted to students from higher income families 
and away from the poor. The higher the cut-off 
level moved — certainly a strong pressure — 
trie more the funds -would resemble general aid. 

! Within school districts such an allocation 
would also shift services to the wealthier stu- 
deht and would result in less intensive services- 
fot the poor. All things considered, there are 
remarkably few reasons for using test scores to 
allocate funds. Compensatory education would 
cease to be a program for helping the poor. 
There may be good reason to have programs for 
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low achievers regardless of income, such as for 
handicapped students, but there is no reason to 
purchase these at the expense of the poor child- 
ren. 

For the most part, I think these studies 
(policy studies rather than evaluations) examin- 
ing the effects of a shift to test score alloca- 
tion, are quite well done. They explore the 
relevant issues in a clever way. Policy issues 
are elusive since they anticipate future ef- 
fects. These studies are conclusive, convinc- 
ing, and succinct. Ordinarily, when one does a 
feasibility study for employing a new method, 
the study is highly likely to favor the new 
method. 1 * That was not the case here. The in- 
vestigators displayed an evenhandedness in deal- 
ing with a critical and politically sensitive 
issue. 

Administration 

Another often neglected aspect of a program 
being evaluated is its administration. The ad- 
ministration of a program cannot only account 
for its efficiency, but can affect its justice 
and fairness. The cornerstone of justice, after 
all, is consistency in application of principles 
and rules. One cannot have a just program where 
regulations are wrongly or inconsistently ad- 
ministered. 

-386- 

31 



The NIE study provided for a thorough ex- 
amination of the Title I legal framework and its 
administration. At the federal level there are 
two types of requirement, those for funds allo- 
cation and those for program development. NIE 
contracted with the Lawyers* Committee for Civil 
Rights Under Law to do a survey of the legal re- 
quirements. The criteria for this assessment of 
the legal framework were whether the regulations 
were necessary to accomplish the program pur- 
poses, whether the regulations were internally 
consistent and consistent with the statutes, 
whether they were clear and comprehensible, and 
whether they were unduly restrictive of local 
district flexibility — an excellent set of cri- 
teria. • t 

The funds allocation requirements included 
seven rules, such as targeting of schools, eli- 
gibility of students, and funds being used to 
supplement instead of supplant. The investiga- 
tors concluded that funds allocation require- 
ments were necessary because of strong pressure 
on states and local districts to use the funds 
in other, noncategorical ways. I would agree 
with this judgment. Without strong stipulations 
the funds would almost certainly be used for 
general aid rather than to help the poor. 

The program development requirements were a 
different matter. They included needs assess- 
ments, objectives, evaluation, and parent in- 
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volvement. While these things may be desirable, 
they are certainly not necessary to accomplish 
the purposes of the program, according to the 
study. In fact, one may meet these program de- 
velopment requirements and still have a low 
quality program. There is little eyidence that 
any of the designated requirements will actually 
result in a better program. I would judge that 
some work against it. There is an unfortunate 
tendency on the part of the government to speci- 
fy not only that something be done, such as pro- 
viding programs for poor children, fcift also to 
fell people exactly how to do it. As often as 
not, the advice # is misguided and ill-conceived. 

There is another difference too between the 
funds allocation and program development re- 
quirements. With funds allocation there are 
strong incentives on the part of the state and 
local agencies to use the funds in other ways 
than those intended. Hence, the restrictions 
make some sense. However, there are no incen- 
tives to deliver inferior quality programs. The 
program development requirements are based on 
the supposition that the federal government 
knows something that the local agencies don't. 
Although some of the requirements are good ones, 
prescribing the contents of local programs' is a 
case of over- regulation. 

As to consistency apd clarity, the NIE in- 
vestigators found the regulations generally /con- 



s latent, although the role of the parent advi- 
sory councils and that of the local and state 
governments in monitoring could be made more 
Specific. On the other hand, the operational 
standards were judged not to be clear. Clarity 
was defined operationally as a Title I adminis- 
trator's being able to comprehend the full range 
of acceptable practices from the regulations. 
State administrators had to invest an inordinate 
amount of time to understand, especially regard- 
ing the requirements of needs assessment, evalu- 
ation, and parent involvement. 

0 

This lack of clarity resulted in overly re- 
strictive interpretations by the states. Be- 
cause of fear of violating the requirements, 
seme states wrongly concluded that they had to 
have pull-out procedures or that they could not 
approve the hiring of nurses, for example. The 
lack of clarity in regulations actually led to 
less flexibility for local districts and to pro- 
mulgation of inferior programs, a point to be 
made later about pull-out procedures. 

At the federal level itself, the investiga- 
tors found that monitoring and enforcement regu- 
lations of the supplement requirement were 
neither clear nor consistent. » Two different 
groups inside the Office of Education charged 
with the monitoring of the supplanting regula- 
tions were in conflict over the proper criteria 
for supplanting violations. As a consequence, 
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the monitoring of this important regulation had 
ceased. States did not know whether or not they 

were in violation. One can see that this kind 

«*■ ♦ 
of information is valuable for an evaluation to 

uncover because something can be done about it 
immediately. 

Overall, the federal management of Title I 
absorbed $5 million and employed 100 people. The 
states spent $20 million for management and the 
It, 000 local districts, $120 million. The state 
agencies received one percent of the state funds 
for administration, and the local districts 
spent about four percent of their funds for man- 
agement. These are not bad figures for such a 
large program. State staffs averaged 18 full- 
time people, although this varied considerably 
form 3 to 101 persons. " Generally, the states 
with poor management performance, as judged by 
such things as audit exceptions, had small 
staffs. The successful state managements spent 
much time with the staff of local programs. 

NIE was also instructed to investigate the 
state compensatory education programs other than 
Title I to see if there -were other models of 
good practice the federal government might 
adopt. This investigation seemed rather cursory 
to me. Fourteen , state compensatory programs 
provided a sum of money equivalent to 20 percent 
of the Title I funds, but 6 8 percent of this was 
in New York arid California. According to the 
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study » the servioes provided by these state pro- 
grams were very similar to those of Title I. 
Even the administration of these programs * was 
conducted from Title I offices. In the opinions 
of the state directors of these programs, Title 
I was essential tc the success of the state pro- 
grams. Few- states would have established their 
own programs without the federal example, and 
few state agencies could resist pressures to 
discontinue. Ironically, ' some of the strongest 
fiscal pressures on the compensatory programs 
(which ruled out increased funding) were federal 
incentives and requirements for handicapped • 
children. 

Some effort, though not a lot, was put into 
studying the administration of local programs. 
At the district level, planning was done in a 
few meetings in the spring. Few teachers or 
principals attended these meetings, and teachers 
did not feel constrained by plans developed 
there. Presumably, these activities were * to 
meet the program development requirements. Plans 
did not relate, to the classroom level. Appar- 
ently the real decisions were made at the school 
level meetings, which involved the participating 
principals and about 90 percent of the teachers. 
In these meetings instructional techniques were 
discussed. But information on this process is 
sparse and fragmentary. 

The elevations required of the programs 
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were not disseminated below the district level, 
and when they were, teachers felt the standard- 
ized test scores provided little information of 
relevance to the classroom. Fully 95 percent of 
the evaluations relied on « such tests, and, in 
fact, the use of such tests is now mandated by 
the Office of Education. Evaluation of compen- 
satory programs continues to be frequently dys- 
functional. It is unfortunate that useful 
evaluations cannot be done. Again, the regula- 
tions specify not only that an evaluation be 
done, but how it should be done, usually counter 
to professional advice. 

Another program development requirement and 
problem area was the Parent Advisory Council. 
The regulations provided that the majority of 
the members must be parents of participating 
children and be selected by parents from that 
area. The .idea was to give poor parents some 
say in the education of their children. This 
was easier said than done. Most councils met 
less, than once a month, and they were purely ad- 
visory 90 percent of the time. In half of them, 
the principals dominated the choices to be made, 
according to -the study. In spite of this, 71 
percent of" the members reported being satisfied 
with ' their role. Some did perceive some un- 
fairness. Again, the NIE account is thin. 

Overall, the evaluation of the management 
of the compensatory program is quite strong, 
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well-considered, and unusual in its focus, es- 
pecially the part dealing with federal regula- * 
tions. This should be an important component of 
similar evaluations. By addressing the concerns 
of those who must live by these regulations, it 
substantially increased the utility and justice 
of the study. The major weakness was a lack of 
similar thoroughness at the state and local lev- 
els. 

i A 

Compensatory Education Services 

* 

The NIE study was actually a series of sep- 
arate studies, more than 35 altogether, which 
were organized and published in 7 separate vol- 
umes. Each substudy addressed a separate set of 
questions. The logic of this design is another 
reason for the success of the overall study. An 
alternative way of proceeding might have been to 
design a massive experiment, the results of 
which would have left unanswered most key ques- 
tions. In other words, a whole set of relevant 
questions was explored by the NIE study, thus 
sustaining and eliminating various hypotheses 
about compensatory programs. Social issues can 
rarely be resolved by a single massive experi- 
ment. There are too many separate issues 
address by a single approach. 

The first volume of the HIE study focused 
on a description of the compensatory programs, 
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especially Title I. This was necessary because 
no current information^' representative of the na- 
tion existed on the selection of students, the 
educational services, or the costs. The multi- • 
tiered reporting system, in which local districts 
reported to the states and the states reported 
to the federal government had resulted- in out- 
dated and- generally unreliable and noncomparable . 
data. The *'NIE survey was based on a nationally 
representative sample of 100 school districts. 
Since unreliability was a feature of past stud- 
- ies, interviews were conducted not only with the 
district directors of the programs,, but also 
with principals, teachers, .and parents witlfin 
each district. Interviews ranged from 30 min- 
utes to two and a half hours. Previous studies 
had relied on self-report, mailed question- 
naires, usually to district administrators . 

The advisability of such expensive ,^ labori- 
ous, face-to-face interviewing may be illustrat- 
ed.by the following anecdote. Between 1967 and 
1971, I directed a large-scale evaluation of the 
Illinois Gifted Program, a categorical program 
that was funded similarly to Title I. In the « 
initial data collection, questionnaires^ were 
mailed to all 1,100 Illinois school districts; a 
respectable return was received. The question- 
naire asked for detailed descriptions of, pro- 
grams for the gifted supported by state funds. 
We analyzed and sent the results to the State 
Department of Education and the State Legisla- 
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ture — we were obliged to provide a report the 
first year. 

♦ 

In the third year of the evaluation t .we 
examined the financial claims of the school dis- 
tricts and found that 31*0 districts* were re- 
ceiving stfate funds. From tliescFwe "selected a 
10 percent random sample stratified by type and 
size of school district, and sent teams of peo- 
ple trained with special interview schedules, 
observation protocols, and psychometric instru- 
ments' into each district. Generally, there was 
little resemblance between the original ques- 
tionnaire description of the local program and 
what our data-collection teams found; 

A few administrators admitted forthright ly 
that their gifted program was a fabrication 
which existed only on paper. I|i other cases, 
the respondents had answered sincerely, but in 
fact had little idea of what their programs con- 
sisted. Only when we interviewed the teachers 
did we discover what- was happening. In even 
more cases, people were misled by our question- 
naire. For example, when they marked "individ-. 
ualized instruction" as a type of program, they 
often meant something quite different from what 
we had supposed by the label. We fooled our- 
selves by our terminology. • All of this misin- 
formation was discovered only when interviewing 
several people and observing the classes within 
the districts. 
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Reliable information is often bought only 
at high cost. In % mjast cases mailed question- 
naires are inadequate as information sources. 
The chances for intentional and unintentional 
misinforming are too great. Fortunately, in- 
depth cfcata collection was what the NIE survey 
Undertook, which makes this part of the study 
seem accurate and credible. They also conducted 
case studies in 18 of the districts (as did we) 
to determine how district auxiliary services 
were provided. The only fault I can find with 
this part of, the study is that they apparently 
did not visit classes. 

Although federal money was awarded to 
school districts on the basis of income level of 
the children's parents, children became eligible 
for Title I services at the school level by be- 
ing identified as having an educational need. 
They were chosen by low test scores and teacher 
judgment. Of those eligible, 66 percent re- 
ceived special educational services. , 

The provision of special services was di- 
rectly proportional to the amount of special 
funding available to the district. The more 
money available, the more students served. There 
was no significant variation in student pat>tic;j.- 
pation rates in districts of different .sizes, 
different regions, or different socio-economic 
characteristics. Children. in non-publi^ schools 
received fewer, services. Of the: students 
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served, 59 percent were' white (compared to 75 
percent total enrollment), 34 percent were black 
(compared to 19 percent of total enrollment) * 
and 10 percent were Spanish-sumamed (compared 
to 5 percent of total enrollment). Contrary to 
popular conception, Title I is not solely, ex- 
clusively, or even primarily for ethnic minori- 
ties. It is for the poor. 

6 

Services Rendered 

What kinds of services were being offered 
to children identified as needy? Describing 
educational programs is difficult because there 
are so many factors that may be significant and 
contribute to or detract from program quality. 
An experienced educator ordinarily has little 
difficulty recognizing a good or a poor class, 
but measuring characteristics that are indica- 
tive of high quality by standard measurement 
procedures is a problem. 

The difficulty is that one might expect a 
small-size class to be a better one, all other 
things being equal. But other things are not 
usually equal, so a small class with a bad 
teacher will result in poor quality education. 
There are so many relevant factors, and they 
interact so strongly with one another that some 
researchers have despaired of being able to pre- 
dict precisely the pattern of factors that will 




result in quality education (Cronbach, 1975). 
One, strategy is to forego description of the 
program and simply measure*-***^ outcomes. But 
the outcome measures themselves, such as stan- 
dardized achievement tests, are often oblique or 
even invalid measures of educational quality. 
This has been the past path to failure. 

What the NIE evaluators did was to identify 
program characteristics that educational re- 
search has suggested are associated with high 
quality. This is not what I would have done, 
but it is not a bad strategy and is certainly 
preferable to attempting to rely exclusively on 
outcome measures. Alternatives might h3ve been 
to conduct in-depth case studies or to assemble 
a larger number of descriptive and evaluative 
observations, both of which would provide a 
denser, thicker description of the programs. 
(See House, Steele, and Kerins, 1971). The NIE 
strategy was a good one, however. 

The factors selected for the study were 
time spent on instruction, class size, teacher 
qualifications, and the extent of individuali- 
zation. The NIE survey revealed that three- 
quarters of the Title I funding went directly to 
instructional programs. Ninety- five percent of 
the school districts offered remedial reading or 
language arts programs. Half had only remedial 
reading or language arts, while 41 percent had a 
math program. Almost all were elementary school 
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programs , 

The average Title I student received 3 
hours and 47 minutes of special instruction a 
-week* if the program was reading, M hours a week 
if language arts, and 3 hours a week if math. 
Frpm 1/M to 1/5 of the students 1 time was spent 
in compensatory instruction. Title I programs 
were deliberately supplemental and in addition 
to regular instruction. The average class size 
was 9 in remedial reading and 12 in language 
arts and math, compared to 27 in regular class- 
es. \ 

As to individualization of instruction, as 
defined by the NIE survey, 3 8 percent of the 
teachers reported using instructional objec- 
tives, 77 percent said they used tests, and 70 
percent reported teaching to small groups or in- 
dividuals. Individualization, one might add, 
could have been defined in several different 
ways. 

Teachers were selected primarily because of 
academic training or experience in compensatory 
education. The average total teaching experi- 
ence of compensatory teachers was similar to the 
national average. The most highly valued train- 
ing, as reported by district personnel, was in 
measurement and evaluation, new techniques, new 
materials, and characteristics of the disadvan- 
taged. Two-thirds of the teachers had received 
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special training, the average being 25 hours. 
Most^districts provided some in-service training 
for teachers, usually consisting of consultants, 
staff meetings, and workshops. Only a small 
portion of the Title I budget (0.5 percent) was 
spent on in-service training, the average dis- 
trict expenditure being $600. 

Most compensatory services were offered as 
ff pull-out ,f programs, that is, students were re- 
moved from the regular classes and put into 
special ones for their compensatory instruction. 
Aboijt 2 5 percent of the students were in special 
classes for the entire day, a feature more prev- 
alent in districts with many eligible students. 
The school districts 1 Title I applications for 
funds had low utility for predicting district 
practices. Most of the instructional expenses 
consisted of teacher salaries, and the number of 
teachers seemed to determine the number of stu- 
dents served and the total time spent in read- 
ing. 

In reading programs relatively more teacher 
aides were employed; in language arts programs, 
there were more teachers with bachelors degrees, 
and more with masters in reading and math. Pull** 
out programs used more aides and teachers with 
masters degrees. In programs with more instruc- 
tional time, there were more aides and less 
well-educated and less well-paid personnel. 
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In 12 school districts, case studies were 
conducted of the programs, but this information 
was confounded by the fact that these were the 
"demonstration" districts which were trying out 
different funds allocation procedures. They 
were not typical districts. A closer examina- 
tion of their programs revealed that the Title I 
students spent 136 minutes a week in language 
arts compared to 122 minutes for regular stu- 
dents. There was a great deal of variation from 
district to district, however. In these dis- 
tricts Title I students spent the same percent 
of time receiving instruction as individuals (15 
vs. 13 percent for ragular students), more time 
in small groups <2«* vs. 9 percent), the same 
time in medium groups (32 vs. 32 percent), and 
less time in large groups (29 vs. 46 percent). 
Small groups were defined as 2-5 students, 
medium as 6-20 students, and large groups p.s 
more than 20 students. Clearly, it was in the 
small group instruction that the Title I 
students had the advantage. 

Only 4.6 percent of the total budget was 
spent on auxiliary services in 1975-76. The 
federal emphasis on basic skills had resulted in 
yearly declines. About 1/3 of this amount was 
spent on parent involvement, another 1/3 on 
guidance and social work, and the other 1/3 on 
health, food, and transportation. Health serv- 
ices consisted of nurses, screening, immuniza- 
tion, referrals, and physical exams. Health and 
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food costs together accounted for only one per- 
cent of the budgets. The Office of Education 
had emphasized that Title I be an educational 
rather than a welfare program. 



The nature of auxiliary services was de- 
termined by OE emphasis away from health and 
welfare, by federal emphasis on parent-advisory 
committees, and by the type of evaluations that 
were conducted. The OE' guidelines demanded 
quantitative measures of educational outcomes, 
and school districts did not know how to assess 
health and other auxiliary services, using these 
measures. Opinions of parents and others were 
considered too subjective. Consequently, dis- 
tricts dropped these services in favor of serv- 
ices they could measure with achievement tests, 
according to the NIE study. . This is a case of 
programs being changed to serve the evaluation 
rather than vice versa, and I would condemn this 
effect of evaluation more strongly than do the 
NIE researchers. 

How good were these compensatory programs? 
The NIE survey provides some criteria by which 
the programs can be judged — if one is aware of 
other educational research on class size and so 
on. First, most money goes into salaries for 
instructional personnel. More money provides 
more instruction for more students. In this 
sense, the benefits are directly proportional to 
the money spent. 
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The classes themselves are small, averaging 
9 students in reading and 12 in language arts 
and. math. From Glass and Smith's *(1979) meta- 
analysis of the effects of class size on 
achievement, one could expect a standardized 
differential achievement of .256, equivalent to 
about 10 percentile points, quite a significant 
gain. That assumes, however, that all the stu- 
dents' time was spent in classes of such size. 
Actually, only one-quarter was so spent. For 
classes of 12 students, one woul'd expect a gain 
of about 5 percentile points. Also the expected 
gain would be slightly lower than this for ele- 
mentary students, which most Title I programs 
service, and slightly higher for secondary stu- 
dents. Other things being equal, tutorials of 
one person compared to classes of 40 students 
could make a difference of 30 percentile points, 
a difference substantially greater than that 
demonstrated for any other instructional tech- 
nique. 

The fact that students in compensatory pro- 
grams are spending more time in reading, lan- 
guage arts, and math than are students in regu- 
lar instruction is also indicative of a high 
quality program since more time on task is as- 
sociated with increased achievement (Rosenshine, 
1978). Unfortunately, the NIE data are not suf- 
ficiently detailed to permit more specific esti- 
mates of gain. 
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Two other factors — individualization and 

* 

pull-out procedures — are more problematic* 
Individualization is variously defined, and the 
research on its effects is mixed. The next sec** 
tion of the review will demonstrate this weak- 
ness. Pull-out procedures in which the children 
are physically removed §£om the classroom for 
special ^ instruction probably have some harmful 
effects, to be discussed in (detail later* 

' • D 'Li 

The information on teachers provided by the 
NIE survey I find useless as far as judging the 
quality of 'the programs. What teachers are 
called makes little difference in quality. If 
an evaluation does not arrive at a defensible 
judgment of worth, it should at least strive to 
provide information that the reader can use to 
make such judgments. That teachers are selected 
on the basis of training and previous experience 
is good, but the training appears to be slight 
on average. One wonders whether the teachers 
are really prepared. Also very little was spent 
on in-service training. One would think that 
there would be far more attention, to improving 
teacher skills in such an important and huge 
program. An average annual district expenditure 
of $600 leaves much to be desired. 

Finally, the federal de-emphasis on. auxil- 
iary services is acceptable rf children are 
provided medical services, referrals, etc., from 
other sources. Correction for near-sightedness 
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is likely to result in far mor educational 
achievement than most things one can think of. 
Especially since medical services are so expen- 
sive, one wonders whether the children are re- 
ceiving basic physical care. The Title I pur- 
poses include concern about comprehensive de- 
velopment of the child. The evaluation should 
have investigated why the Office of Education 
has de-emphasized auxiliary services and whether 
the children were being taken care of elsewhere. 
This is one of the serious deficiencies of the 
study. 

To be fully assured that the compensatory 
programs are of high quality, one would have to 
investigate the classrooms in more depth. This 
was only one aspect of the study, and one cannot 
do everything. Yet, one longs for a deeper un- 
derstanding of the compensatory classrooms, the 
"complete understanding" aspired to by the eval- 
uators themselves. Nonetheless, I believe the 
NIE evaluators had their priorities essentially 
correct as to where to put their resources. 

7 

Student Development 

The third fundamental purpose of compensa- 
tory education, according to NIE, was to con- 
tribute to the cognitive, emotional, social, or 
physical development of the participating stu- 
dents. The NIE substudies which addressed this 
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purpose were more typical of the way such evalu- 
ations are usually done, and less satisfactory 
than the rest of the NIE study. The evaluators 
rejected the notion of a grand national summary 
evaluation of Title I because. £the past evalua- 
tions had been failures. These' failures had 
occurred, they said, because of the diverse na- 
ture of the programs, the masking effects of a 
grand summary measure, and the limitations of 
measurement instruments. Instructional programs 
are not unique, nonover lapping treatments. This 
was good thinking on their part. 

Instead , they chose a sample of districts 
to investigate. The sample was selected to pro- 
vide a broad array of instructional variables 
that could be related to achievement gains* 
Generally , studies that have attempted to relate 
quantified instructional variables to achieve- 
ment gains have not fax^d well. The main vari- 
ables here were instructional time and content, 
amount of individualized instruction, and the 
difference between instruction in regular class- 
es and pull-out classes — classes in which Ti- 
tle I students are withdrawn from the regular 
classroom for instruction. 

The Comprehensive Test of Basic Skills was- 
administered to students in U00 classrooms in 14 
different districts in the fall, spring, and 
following fall. The sample jwrif^nqjt chosen to be 
representative of Title I classes as a whole, so 
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the results cannot be generalized to all Title I 
classes. The investigators did claim consider- 
able diversity of programs in the sample. 

From fall to spring, a" perijod of seven 
months, students in the first grade gained 12 
grade equivalent months in reading and 11 months 
in math. Third graders gained 8 months in read- 
ing and 12 months in math. The average pupil in 
the norm group gained 7 months in the same time.* 
These gains were . quite high compared to o%hev 
evaluation studies of compensatory education. A 
follow-up study in the fall found the students 
holding, these gains on average, a finding again 
ccfhtrary to expectations. 

One must wonder about the sample of stu- 
dents and the test. On the pretest the first 
graders averaged at the 22nd percentile in both 
reading and math, and the third graders averaged 
at the 13th percentile in reading and the 5th 
percentile in math — two quite different groups 
of students. Besides having no experimental 
controls, the same tests were administered to 
the students on three occasions, providing op- 
portunity to learn the test s d/bems.. The results 
are strikingly positive but not convincing as to 
the effects of Title I programs. . $ 

1 

Glass and Smith (1977) questioned these re- 
sults. The pretest raw score mean for first- 
grade reading was 2.38, close to the chance raw 
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score of 24.8. The first-grade math raw score 
mean was 15.6, about three points below the ex-* 
pftcted mean of 18.7 if students guessed random- 
ly. This raises the specter that Level B of the. . 
Comprehensive Test of Basic Skills was too dif- 
ficult, to use as a pretest for the first grad- 
ers, and .that the achievement gains were spuri- 
ously high. This could account for the unusualj- 
ly positive results. This t suspicion is 
strengthened by the low pretest and post-test 
correlations on the same test for first grade 
(r = .50 in reading and .39 in math compared to 
.86 and .78 in third grade). 

As in most such studies, the focus of meas- 
urement was narrow — only reading and math at a 
few grade levels. The evaluators felt that 
there was inadequate instrumentation to assess 
the other areas of student development. In this 
they were probably correct, but nonetheless 
judgments about the program were made on this 
narrow basis. Interviews with students, par- 
ents, and teachers would have been valuable in 
assessing development. The researchers report 
l in a footnote that student attitudes were high 
in the fall and the spring but give no further 
information about this aspect of the study. 

How these instructional variables related 
4 to gains in test scores was also ambiguous. The 
first graders gained more in the regular class- 
rooms, in both reading and math, but the third 
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girders in the pull-out classes did better in 
math and the same in reading. Was this because 
th£ students were different or because of spuri- 

;ous measurement? One thing is clear: student 
gains were large when the instruction emphasised 

1 the skills measured by the test. It was possi- 
ble *td measure the overlap with the curricula by 

'analyzing the test items and the video tapes of 
classes, instructional materials, and interviews 
with teachers. This was an important precaution 
that is seldom taken, ana one of the best fea- 
tures of the study. 

Classroom processes were ' measured by indi- 
cators of opportunity to learn, individualiza- 
tion, "instructional events," motivators, and 
* teacher background. Taken as a whole, these 
classroom processes accounted for a significant 
amount (.1U to .27) of the variance in achieve- 
ment gains in first and third grades. In this 
analysis the evaluators quite properly used the 
classroom rather than the i-fffflvidual student as 
the unit of analysis. These figures and others 
are taken from a later and more detailed com- 
mentary than that provided in the original re- 
port (Cooley and Leinhardt, 1980). 

When the five classroom process indices 
were taken separately and in combination, only a 
few accounted / for modest proportions of the 
achievement gain. "Opportunity" accounted for 
.22 of the variance in first grade reading, .09 

-»f09- 

54 



\ 

in math, .09 in third grade readkng, and .13 in 
third grade math. Most contributions were in # 
the .range of .01 to .0*» of the variance in test 
gain. 1 

When one looks more closely at the "oppor- 
tunity to learn" index, it is dominated by the 
overlap measure between the test and curriculum. 
This is a combination of two scores, one an es- 
timate by teachers of whether they had taught 
the test items and the other based on an analy- 
sis of the curriculum by experts. Zero-order 
correlations of this variable with the test gain 
were .U0, .25, .29, and .38 (in reading and math 
at first and third grade). Clearly, test over- 
lap accounted for a substantial portion of the 
"opportunity" index, i Furthermore, when one ex- 
amines all the other single variables, the 
highest correlations were with "time opportun 
ity," "student mastery," and "matching" in vari- 
ables. The latter two measure test-taking prac- 
tice in the curriculum. 



The investigators concluded that what is 
taught is more important (in affecting test 
scores) than how it is taught. Children will do 
\ mu *h better on tests if they are exposed to both 
the form and content of the test itself. The 
researchers recommended an intensive examination 
of the match between the content of- curricula 
and tests. Future evaluation,* they suggested, 
should contain a measure of curriculum fit to 
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the test. 

I agree with this recommendation but would 
turn their suggestion around. The most parsimo- 
nious, explanation of these results is not that 
the curricula are in error somehow, but that the 
tests are. They are far too narrow in scope. 
If one insists on evaluating curricula with 
narrowly focused tests, the results will largely 
reflect the degree to which the curriculum 
teaches these particular test items. One can 
indeed develop a curriculum to teach those test 
items and do better on the tests, but such edu- 
cation would be strange indeed, and perhaps 
worthless. It is an inverted logic which in- 
sists that schools teach what standardized tests 
measure. One can envision an entire society 
giving itself tests and succeeding on them as it 
becomes progressively removed from reality and 
experience. 

The main finding of the study, as seen by 
the investigators, was that classes with indi- 
vidualized instruction (as defined in the study) 
were no better than those without. The investi- 
gators expressed the opinion that there was a 
low degree of individualization in most classes, 
in their opinion. Apparently, the Congress was 
considering mandating such techniques in 19 74 
and was deterred by the lack of positive evi- 
dence about individualization here and else- 
where. This is another mandate that we are far 
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better off without, and the study was certainly 
valuable if it helped prevent such an action. 

Overall, in spite of some interesting ideas 
about how to measure and analyze instructional 
variables in a way closer to classroom experi- 
ence, the study did not provide a very full or 
convincing picture of student development in 
compensatory classes. Gains in the classroom 
seemed to be related to certain activities, but 
one can't say much more. The study becomes one 
more equivocal essay of its type. It is more 
suggestive than persuasive. The results are not 
satisfying. 

Because of the flaws, limitations, and mix- 
ed results, the NIE study is also inconclusive 
about pull-out programs. This is an important 
issue for compensatory programs because the ma- 
jority of Title I students received instruction 
in pull-out classes. Fortunately, the Office of 
Education commissioned a policy study by Glass 
and Smith (19 77) on the issue, which proved to 
be more persuasive. Although this study is not 
part of the NIE study, the issue is so important 
for Title I that I mention it here. 

To address the issue, Glass and Smith con- 
ducted an analysis and meta-analysis of research 
studies from related areas such as ability 
grouping, mainstreaming, and desegregation. Com- 
bining these studies, they found that labeling 
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students as "slow" or "weak" reduced their aca- 
demic performance by one-quarter standard devia- 
tion below that for comparable pupils not so la- 
eled. Teacher attention and support for the pu- 
pils was reduced by one-third standard deviation 
and teacher judgment of student success by a 
half standard deviation. From their analysis 
"the pull-out procedure per se has no clear aca- 
demic or social benefits and may, in fact, be 
detrimental to pupils* progress and adjustment 
to school." (Glass and Smith, 1977, p. 7). This 
study is more persuasive because it relies on 
the combination of many strands of research 
rather than attempting to base its conclusions 
on one study with all its idiosyncracies and 
weaknesses. Also, in this case the findings 
from various areas are consistent in indicating 
that the pull-out procedure might be detrimen- 
tal. The dependent measures are all cognitive, 
however. 

Glass and Smith took the position that I 
advocated earlier: all pupils in poor schools 
deserve compensatory services. Any child in a 
school with a high concentration of poor pupils 
is put at an educational disadvantage. By aim- 
ing funds at schools with large concentrations 
of poor children, the beneficial effects of 
categorical aid could be retained and the in- 
vidious effects of pull-out procedures avoided. 
Only a matter of administrative convenience in 
monitoring has led to the pull-out practice. In 
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fact, the management of the program would be 
better if the unit were the school. 

This is an acute case of officials being so 
concerned that someone ineligible might benefit 
that those who rightfully should benefit are de- 
prived, a case of monitoring working against the 
intent of the program. While the zealousness of 
concern for the poor is laudatory, one must be 
careful when pursuing such policies that they 
are net counter-productive. Professional judg- 
ment, research, and common sense argue against 
the pull-out procedure. 

In this particular case, the OE study sue- 
ceeded where the NIE study did not. However, 
the overall effects of these compensatory pro- 
grams on student development remain unassessed. 
Educational measurement continues to be inade- 
quate to the task of assessing braod student de- 
velopment , there are no magic solutions; evalua- 
tors should be more honest about what they can- 
not do. 

» 

Summary 

From reviewing the NIE compensatory educa- 
tion study, I have two strong impressions. One 
is that the compensatory programs, especially 
Title I, are very good programs. The second 
impression is that it is possible to evaluate 
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such programs intelligently, usefully, and val- 
idly. That these enterprises were the respon- 
sibility of the federal government, is striking, 
for the federal performance on these matters has 
not been good over the past several years. 

My impression of Title I is that it is a 
categorical program that has worked. The pur- 
pose was to help the children in low income 
districts, and they are being helped. Funds go 
to the regions, districts, cities, and schools 
with lowest incomes. Compared to other federal 
and state programs, it is perhaps the most re- 
i distributive of all, an impressive accomplish- 
ment. 

Of course, it will not eradicate poverty. 
The resources are too meager for that, and prob- 
ably poverty stems from quite different sources 
than the schools. Eradicating poverty is too 
much to expect. The funds are enough to make a 
discernible difference in the education of the 
children in poor areas, however. Other methods 
of allocation would be less equitable and fur- 
ther removed from the purposes of the program, 
as well as less accurate ahd less efficient. 

I am persuaded by the NIE study that most 
of the funds allocation requirements are nece&r 
sary and not overly restrictive. The program 
development requirements, on the Other hand, 
edge toward over-regulation and the mandating of 
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practices for which there is little supporting 
evidence, scientific or professional. This 



seems to me to be an all too typical misguided 
foray" into prescribing practices that experi- 
enced people on the scene would not endorse* 

Unfortunately f many government p lanners 
cannot seem to tell the difference between re- 
quirements that are necessary because local in- 
centives work against them, as with funds allo- 
cation, and requirements that are detrimental to 
c sound educational practice, which is determined 
by experience as in any craft. The worst ex- 
ample is the mandating of particular types of 
evaluations that are contrary to the expertise 
of the evaluation community. 

I am persuaded that the management of the 
compensatory program at all levels is reasonably 
efficient and effective, although I suspect that 
much of the time of state and local administra- 
tors is spent meeting requirements that have 
little to do with classes for children. I sus- 
pect that the real job is done by extra person- 
nel at the school and classroom level who do 
what they can. Most of the funding supports 
these extra personnel. 

The monitoring of these programs appears 
confused and conflicted because of the impasse 
on criteria for monitoring inside the Office of^ 
Education. This situation might have been par- 
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tially resolved, and the monitoring made easier, 
by making entire schools eligible ^n low income 
areas. Such a policy might have countered the 
apparently deleterious effects of the pull-out 
procedures, which were instituted for. fear of 
violating unclear and ambiguous federal guide- 
lines. Overall, the programs appear to be of 
high quality, primarily because of the employ- 
ment of more personnel. They focus on the bas- 
ics in reading and math. Exactly what they con- 
sist of is not clear, though one might question 
the effectiveness of pull-out procedures and in- 
dividualized instruction. 

*\ 

I am not persuaded as to how students are 
developing in and as a result of these programs. 
Although the test score gains were very impres- 
sive over a one-year* period in the sample class- 
rooms, there are a number of problems with the 
study of student development. In addition, even 
if the gains were real and indicative of oyerall 
quality, the aspects of development investigated 
were narrowly construed.* Both the exact nature; 
of the compensatory classrooms and the specific 
development of children in these classes t remain 
hazy. 

For better or for worse, I have gathered 
these impressions primarily from the MIE study. 
I have had little first-had experience with* 
these programs or classes. The persuasiveness 
of the NIE study is partially dependent on its 
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validity. It is well-conceived as a series of 

substudies addressing a large number of issues. 

t 

In its intensive investigation by inter- m 
view, observation,' and inspection, it seems to 
be accurate. Loose ends are pursued, alterna- 
tive hypotheses examined. In other words, the 
study is generally conceptually well-reasoned. 
The possible exception here is the work on stu- 
dent development, which leaves a number of ques- 
tions unanswered. There are too many flaws for . 
these conclusions to be persuasive. 

The NIE Study as a whole is also credible. 
Its enthusiastic reception by Congress is one 
indicator ofJets relevance and credibility to 
its major andience. It was done by an agency 



external tro the program and by different re- 
searchers and groups, many of them well known. 
The results are also internally consistent. The 
final .report could have been more coherent by 
drawing the many disparate strands together. 

Finally, the evaluation was just and fair 
and democratic. A just evaluation should attend 
to the interests of all groups, particularly to 
those who are most affected by the programs be- 
ing examined. In this case in particular the 
study should attend to the interests of the poor 
and disadvantaged. The NIE study accomplished 
this by examining in great detail how the re- 
sources and services were distributed. Justice 
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is concerned with distribution, with whoV gets 
what. Through examination of the program admin- 
istration, the investigators also determined how 
recipients and participants were treated. Con- 
sistency and the quality of these relationships 
were important concerns. Of course, in hind- 
sight, all of these things could have tfeen im- 
proved, but in my opinion they were remarkably 
well handled in this study. 

I believe the foregoing fundamental consid- 
erations — the truth, credibility, and correct- 
ness of the study — account for its validity. 
Validity in turn means that the evaluation is 
acceptable and that the audience, Congress in 
this case, can act upon it. Of course, Congress 
may not apt upon it. There may be reasons, le- 
gitimate or otherwise, that prevent Congress 
from acting. 

Because an evaluation is valid and accept- 
able does not mean that it is accepted. Perhaps 
key Congressional leaders know that another fund 
allocation scheme would benefit their own 
districts more than the scheme favored in the 
study. The self-interest of their district 
might override their concern for the poor. Or 
perhaps some important factor changes while the 
study is in progress — the funds disappear. Or 
perhaps the study is lost someplace in the vag- 
aries of the legislative process. For various 
reasons the study may te valid and acceptable 
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but not used* j 

If the study is used, however, there is 
normally a presumption of its validity. Cer- 
tainly the Congress does not examine the implic- 
it validity claims as I have done. Rather, as 
expressed earlier, its concern is not with tech- 

9 

nicalitie's- but with whether the information can 
withstand a professional critique and give it 
some guidance as to what to do. The Congress 
presumes validity. 

It might also be in highly politicized sit- 
uations that people cynically use only that in- 
formation which will benefit their cause. Even 
in thip circumstance, however, the information 
is only effective if the people on whom it is 
used beli< /e that it is valid. It is not enough 
that an evaluation be used; it must be valid. 
And ordinarily it will not be used unless it is 
at least presumed, rightly or wrongly, to be 
valid. 

* In this case there was no problem. The N IE 
study was valid, according to standards I have 
enunciated, and it was used,£>y the Congress to 
very good effect. In this case it served the 
Congress well and was a good evaluation from 
both a Congressional and a professional evalua- 
tor's point of view. 
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FOOTNOTES 

1. Hill pays special tribute to the help 
of David Wiley and Jim Vanecko in the conduct of 
the study. 

2. * Unless otherwise noted, all facts and 
figures about Title I are taken from the HIE 
study, . which was published in seven separate 
volumes. These figures are from Volume 2. 

3. The information in this section is 
primarily from Volumes 3 and 4. 

Congress did eventually" reject the 
notion of allocation by test score. One factor, 
according to Hill, was that the study did not, 
permit the estimation of allocations for indi- 
vidual Congressional districts. Not knowing 
what their districts would receive under a new 
allocation scheme, Congressmen retained the old 
allocation procedure. . 

5. The information in this section is 
from Volume 5. 

6. The information in this section is 
* primarily from Volume I . 

7. The informatiqn in this section ia 
primarily from Volilmes 6 and 1. 
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8. The Title I programs have undergone 
substantial changes since these studies -were be- 
gun in 1974, partly as a result of these. stud- 
ies. I an not well informed enough about the 
current program to suggest exactly what these 
changes have been. 
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Evaluating Compensatory Education 

/ 

Torsten Husen, University of Stockholm 

Particulars About the Point of Departure 

I have in my office four big files of news- 
paper clippings from the New York Times , the San 
Francisco Chronicle , and the Palo Alto Times on 
educational issues at the national, state, and 
local level collected during the academic year 
1965-66 which I spent at the Stanford Center for 
Advanced Study in the Behavioral Sciences. I 
tried to- take advantage of my year at this 
"think-tank" to deepen my knowledge of American 
education, a process tjhat had started more than 
ten years earlier whenj I visited half a dozen 
universities and later! spent some time as a, vis- 
iting professor at thei Graduate School of Educa- 
tion at the University of Chicago. One of my 
thick files contains clippings from 1965^66 on 
Head Start, urban education, segregation, van- 
dalism, education of t^e minorities, etc., in 
short, social-pedagogical problems. The arti- 
cles emanate from the heydays of the new legis- 
lation of the Johnson administration on , compen- 
satory education passed \ under the auspices of 
the Great Society. 

I did not envisage, when this material was 
collected, that it would prove useful in provid- 
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ing a time perspective for the review of U.S. 
\federal policies for the education of the dis- 
advantaged launched by the Organization for 
Economic Cooperation and Development (OECD) in 
Paris as one of the so-called country reviews on 
educational policies that OECD sponsors in co- 
operation with the government whose policies are 
under scrutiny. Nor did I envisage that I 
should be serving as the rapporteur of the re- 
view team which implied that several additional 
months had to be spent both before and after the 
site visits on covering, hopefully, relevant 



documents and research literature and, of 
course, writing the report. The latter is now 
available in print (OECD, 1980). 

The present review has been commissioned to 
focus on the NIE reports requested by Congress 
in 19 74. I shall, however, view the reports in 
the wider perspective provided by, among other 
things, the experiences gained during the OECD 
policy review. Reference will also be made to 
some of the 35 projects which form the basis for 
the six NIE reports listed above and which in 
addition to the interim reports form the basis 
for the final report on compensatory education 
that NIE submitted in September 1978 shortly be- 
fore Congress was about to take final decisions 
on a 5-year re-authorization of the various pro- 
grams, and in particular about those under ESEA, 
Title I. 
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Since the OECD policy review provides the 
framework for my attempts to review the NIE 
studies of compensatory education programs, it 
would be in order to give a brief account of how 
this exercise was carried out. In the case of 
the United States it was on the whole conducted 
according to the same procedure as other OECD 
country reviews with the important difference 
that it was limited to the education of the dis- 
advantaged. Two previous country reviews, 
France in 1969-70 and the Federal Republic of 
Germany in 1971-72 (OECD, 1971 and 1972), in 
which I also participated, covered the entire 
field of national educational policies. The 
following steps can be distinguished in a coun- 
try review procedure: 

1. The initiative to conduct a policy re- 
view can be taken by the government of a partic- 
ular country which wants its policies to be 
critically scrutinized by outside people not en- 
tangled in its own internal political contro- 
versies and who are free to raise issues which 
can not easily be raised within the country. 
The OECD secretariat can also initiate a review 
of policies in a particular country, which is 
considered to be of common interest and benefit 
to the member countries. Once the decision has 
been taken to conduct a review, the OECD secre- 
tariat in contact with the Government concerned 
appoints a team of reviewers, as a rule per- 
sons who are not only knowledgeable about the 
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educational systems of the country but also 
carry a reputation in the international com- 
munity of scholars that lends sufficient weight 
to their work. 

2. The government under review prepares a 
National Report which essentially is an attempt 
to collate relevant legislative, statistical, 
and research information pertaining to its edu- 
cational system. In the case of the U.S. review 
of compensatory education the U.S. Office of 
Education (OE, 1979) prepared six extensive case 
studies of compensatory education programs, such 
as Title I, the bi-lingual programs, the basic 
Educational Opportunity Grant program, etc. The 
reviewers received in advance piles of legisla- 
tive, executive, and research documents. The 
task even to cover the most relevant ones was 
enormous, particularly for the rapporteur who in 
the U.S. case, assisted by Dr. Lawrence Saha of 
the Australian National University, had to pre- 
pare an issues paper. A few examples will il- 
lustrate the gigantic documentation problem. 
The review of Head Start research since 1969 and 
the bibliography that goes With it commissioned 
by HEW and published in 1978 is a 158- page docu- 
ment (Mann et al., 1977). The Abt Associates, 
Inc. evaluation $f the Follow Through Planned 
Experiments is reported in seven volumes, etc. 
In addition, HEW through the Office of Education 
submits a voluminous annual evaluation report to 
Congress on its programs. 
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3. Site visits * and interviews with key 
people dn the country under review are conducted 
by the entire review team. In the case of the 
U.S. the team spent about *ohe week interviewing 
people in Washington, D.C, beginning in the 
White House, proceeding with the Office 6f Edu- 
cation and other Federal agencies and with the 
relevant Congressional committees. It also 
tried to meet representatives, as many as pos- 
sible, of the countless "advocacy groups" that 
are based in Washington, D.C. We then traveled 
to li different places in nine states and had a 
wrap-up session in Washington, D.C. This was 
what we were able to achieve in a 3-4-week per- 
iod. The rapporteur spent another four months 
working on the report, being during that period 
based at the National Humanities Center in Re«r 
search Triangle Park, North Carolina. 

4. The reviewers at the wrap-up session 
decided on a series of issues that they wanted 
to raise with the representatives of the Federal 
government. After a few months a preliminary 
report was made available to the partners in- 
volved along with a list of major issues broken 
down in specific questions. The report and the 
section on issues formed the background ' document 
for a "confrontation meeting" at the OECD head- 
quarters in Paris under the auspices and pres- 
ence of the OECD Education Committee consisting 
of representatives of the 24 member countries. 
The discussion as usual took place in a kind of 
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seminar setting with one reviewer introducing 
one major issue or question at a time, whereupon 
in this case the U.S. team representing the Fed- 
eral government responded. Members of the Edu- 
cation Committee can and actually did also take 
the floor and posed questions to the responding 
team. The report, together with an account' of 
the proceedings at the confrontation meeting, is 
then printed in English and French. 

Again, in preparing the present review of 

* 

the NIE reports on compensatory education I have 
quite naturally drawn, hedvily on the parts of 
the OECD report which I wrote up. The more 
technical aspects of the evaluation studies were 
for the most part not considered in the OECD re- 
port but have of course been included here. 



The NIE Evaluation Exercise 

In connection with the passing of the 1974 
ESEA Amendments, the 93rd Congress requested NIE 
to conduct a comprehensive study of the compen- 
satory education programs. The Institute was 

instructed by Congress to "undertake a thorough 

» 

evaluation and study of compensatory education 
programs, including such programs conducted by 
States and such programs conducted und.er Title I 
of the Elementary and Secondary Education Act of 
1965." The Institute was specifically instruct- 
ed to conduct 
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"(1), an examination of the* f undamentjal pur- 
poses of such programs, and the effectiveness of 
such programs in attaining such purposes; 

(2) an analysis of means to identify ac- 
curately the children who have the greatest 
needs for such programs, in keeping with the 
fundamental purposes thereof; 

(3) an analysis of the effectiveness of 
the methods and procedures for meeting the edu- 
cational needs of children, including the use of 
individualized written educational plans for 
children, and for programs for training the 
teachers of children; 

(4) an exploration of alternative methods, 
including the use of procedures to assess educa- 
tional disadvantage, for distributing funds un- 
der such programs to States, to State education- 
al agencies, and to local educational agencies 
in an equitable and efficient- manner , which will 
accurately reflect current conditions and insure 
that such funds reach the areas^of greatest need 
and are * effectively used for such areas..." 
(NIE, 1977f, p. k2). 

In- order to achieve the purposes under the 
four clauses cited the Institute .was instructed 
to carry out /'not more than 20 experimental 
programs" which could be commissioned to agen- 
cies eligible for grants. 
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NTE in accordance with the legislative pro- 
vision in 19 77 submitted six interim reports 
(NIE, 1977 a-f) and a Final Report (NIE, 1978) 
to Congress in September 1978 shortly before the 
Education Admendments were on the floor* 

This was, indeed, an overwhelming charge 
typical of the aspirations held by policy makers 
when turning to researchers. As I have spelled 
out in another connect ion ( Husen , 19 7 8), the 
problems as formulated by policy makers must be 
broken down in manageable units in order to be- 
come researchable. Therefore, NIE apparently 
has tried to narrow down the package of tasks 
without deviating too much .from the letter of 
the law. It has attempted to^ restrict its 
charge in two respects. "Iri response to this 
request, NIE implemented a study of compensatory 
education programs, focusing on Title I of the 
Elementary and Secondary Act of 1965 w , is the 
laconic interpretation that NIE makes in the in- 
troductory section of its final evaluation re- 
port (NIE, 19 78). But within Title I there was 
a limitation to "four major study areas": funds 
allocation, service delivery, student develop- 
ment, and program administration, all technical 
in nature and amenable to down-to-earth empiri- 
cal studies. One should, for reasons just giv- 
en, not criticize NIE for opting for such limi- 
tations. Already the task it decided to under- 
take was enormous. Notwithstanding this, one 
nevertheless lacks the wider perspective on the 
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compensatory education programs that the OECD 
review team, given its wider terms of reference, 
was free to adopt. Thus, in the reports one 
does not find any considerable concern with the 
"fundamental purposes' 1 envisaged by the Law. 
Apart from the immanent difficulties stemming 
from the breadth of such a task, one should con- 
sider its political sensitivity. To question 
basic purposes and strategies is to place one- 
self in the middle of political controversy. The 
limitation to Title I can, of course, be defend- 
ed on the ground that the overwhelming portion 
of the compensatory education funds goes into 
Title T programs. 

The six interim reports, as well as the ma- 
jor portion of the final Report, are based on 35 
research projects conducted for NIE by various 
research organizations, such as Abt Associates, 
Inc. , National Opinion Research Center, and the 
Syracuse Research Corporation. 

Even given the considerable restrictions 
the NIE has imposed on itself in conducting the 
Compensatory Education Study under congressional 
instruction, its reports cover a wide range of 
problems pertaining to program administration, 
service delivery, fund allocation, and student 
development. In reviewing the study I shall not 
focus on technicalities but take the liberty of 
applying the u same broad, policy-oriented per- 
spective as did the OECD team and thereby drfcw 
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vheavily on my experiences from this review as 
well as on the/material I prepared as rapport- 
eur. Considering the audience for ,which the 
OECD report was prepared, the technical and 

^Wholarly aspects of the execution of the Com- 
pensatory Education Study were dealt with- brief- 
ly and cursorily only. In the present review 
they. take a more prominent place. However, it 
would be highly presumptuous even within the 
space allotted to the present review to claim a 
thorough critique focusing on the massive ef-v 
forts made in the^arious conunis^ioned projects' 
launched to evaluate specific aspfects of the 
Title T programs. In addition to the 35 pro- 
jects, on which the NIE Compensatory Education 

"Study in the first place has been based, there 
is an enormous scholarly literature pertaining 
£%^to compensatory education programs of various 
kinds. f\ 

I shall not review the NIE Reports one by 
one but m£ke casual references as I move along. 
After having "established my credentials", by 
indicating- what kind of background I bring to 
the review task, I shall as a background present 
the main f^Mrures of .the compensatory education 
legislation. A l^rgfe portiqji of my review will 
focus on what the legislators referred to as the. 
"fundamental purposes", that is to say, the 
philosophy and strategies behind compensatory 
education. On the basis of such an analysis I 
shall discuss the targeting of Title I funds, 
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both its horizontal and veVtica} coverage. Fi- 
nally, ' I shall deal with the 7 massive evaluation 
efforts that have been conducted particularly of 
,the Follow-Through Planned Variation Experi- 
ments. - * 

-...I have already mentioned the conspicuous 
lack of studies of the "fundamental purposes" 
and hastened to indicate that this is something 
that a government agency, such as the NIE, could 
not easily deal with. It would rather have been 
the task for an institution, such as the 
National Academy of Education. Within the 
restrictions that NIE has set for its research 
efforts and — not least — within the limits of 
wnat it could reasonably be expected to achieve 
within the given time-frame and resources, its 
Compensatory Education Study has considerable 
merits. One has succeeded in collecting rele- 
vant research either already conducted or com- 
missioned to outside agencies and in conducting 
in-house Studies. All these together present 
what is essentially a' good descriptive picture 
of how Title I .compensatory education is* oper- 
ating and what its main technical and adminis- 
trative problems are. 

The report (NIE, 1977 f) on the use of 
* achievement test scores to allocate Title I 
funds presents a sound empirical basis as well 
as a good analysis of the technical problems in- 
volved in switching from census-based poverty 
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criteria to achievement criteria. Not least the 
financial implications, particularly important . 
to legislators , have been well spelled out. It 
is, however, difficult to avoid the impression 
that the difficulties that have to be overcome 
in a switch to achievement- criteria have been 
given stronger .* emphasis than the disadvantages 
of- the present system* The mistargeting and its 
conceptual determinants £ad deserved more 
thorough studies/ Since congressmen eagerly 
watch how changes in allocation criteria affect 
the>r constituencies, findings that too con-, 
spicuously reveal inadequacies of present prac- 
tices are easily felt as stepping on their toes. 
The vested interests in status quo in a multi- 
billion enterprise, such as compensatory educa- 
tion programs , are indeed strong- "Congress did 
in its 1978 Education Amendments not make any 
considerable change in previous practices of al- 
locating funds." The only major changes enacted 
were those dealing with the administrative pro- 
cedures, which were simplified, hopefully on the 
basis of findings from the NIE study of adminis- 
tration of compensatory education (NIE, 1977 a) . 



Strategies for Compensatory Education 



The ' role. Americans traditionally have as- 
signed to education is that of aHTehicle for up- 
ward 1 social mobility. 1 Education has more 



; -436- " 

81 



/ 



strongly than i„ my other oountry been 

can D " ^ In8trU ~ nt ° f ««"^ng the Ameri- 

ed"c ST' C ° UPled " lth th « W« in 

"J " ' Pr<M, ° t0r « f '"dividual ca„ers 
has b« e „ the confidence in it as a neans Qf 

solving social problems by „aki„ g p,„pi e „ore 
competent and knowledgeable H " „ ! 
Ar ... "geaDie. Horace Mann, one 

of the architects of the American public school 

conceded this school as a tool of allevla^s 
inequalities' and the disad„»„+ """viating 

disadvantages of the work- 
ing classes. He wrote i„ m „ - Educatlon 
beyond all other devices of human origin Ts £ 
I~.t equalizer, of the conditions of „ e „ - the 
balance wheel of the social machinery." < quot e d 
after Hechinger, .1976). The confidence in edu- 
cate . oatalyst . n changing sooiety wm 

hind "e progressive movement in education be- 

Count s reconstruotiyism was the concept of edu- 
cation as an instrument in bringing about social 
reform. 

We belief i„ education as a means of re- 

ne^am-a " ^ ™' 
Great , " UBd " 1 *». behind the 

Fo" e :: ty le8islatlon - "» Resident's Task 
Force on Education, chaired by Jo hn W. Gardner 
seems to have played an important role in p "'- 
Paring subsequent legislative programs for com- 
pensatory education of the disadvantaged. Not 
only did the Task Force confirm an adherence to 
the traditional tenet by saying that it would be 
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a major task for { the Nation to "provide every 
cMld with as miich education as his talent and 
eriva warrant 11 , but it also recommended steps to 
be taken to give children of disadvantaged back- 
ground access to normal educational facilities. 
The Task Force ' pointed out the great inequali- 
ties between the States and localities in paying 
for education and even recommended general Fed- 
eral aid to school education on the basis of 
some equalization formula, taking, however, a 
dim view of its "political feasibility" (Report, 
196it). 

The Presidential Task Force pointed out 
that the American school so far had done well 
with children who belonged to the mainstream but 
had neglected those at both ends of the spec- 
trum, the exceptionally talented at one end and 
the poor and physically or mentally handicapped 
at the other end. It pointed out that the hand- 
icapped were left out of the picture with regard 
to Federal support and that it would be a Feder- 
' al obligation to take care of the "poor child- 
ren. .. (who^ .are to be found in our rural arid 
urban slums, and these slums breed conditions 
that do in fact diminish the teachability of the 
child" (ibid.). 

\ 

The Task Force recommended two educational 
strategies by means of which one should try to 
come to grips with the poverty problem. First, 
since childreA from poverty areas already at 

-U3 8- 



ERIC 



o3 

0 



V 

\ • 



school entry had fallen behind their age mates, 
experiments with pre-school programs would pro- 
vide stimulation that could make up, for environ- 
mental deficiencies. Secondly, work-training 
programs could be launched at the high school 
level. 

The Federal government's commitment to what 
was called compensatory education began with the 
passing of the Vocational Education Act of 1963 
and the Economic ■ Opportunity Act; of 1964. The 
Civil Rights movement rapidly gained momentum 5n 
the early 1960's at a time when social scien- 
tists began to explore the "other America". The 
Head Start program focusing 0.1 pre-school child-, 
ren in the form of enrichment programs was p|art 
of the overall anti- poverty program. / 

The big breakthrough of Federal commitments 
to the education of the disadvantaged was the 
passing of- the Elementary and Secondary Educa- 
tion Act (ESEA) of 1965. In signing it, the 
President commented: "No law I have signed or 
will ever sign means more to the future of Amer- 
ica." High hopes were indeed held about what 
education could do in rectifying injustices and 
inequalities in American society. The legisla- 
tion embodied in ESEA could also be regarded as 
a breakthrough for "categorical" Federal aid on 
a large scale to local school agencies, although 
the National Defense Education Act of 1958, aim- 
ing at upgrading science, mathematics and tech- 
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nology, presaged further major thrusts. Since 

the American Constitution lacks specifications 

> of Federal responsibility for education, i% has 

by default bpen concluded that support of public 

education is a prerogative of the States and the 

localities* The role of the Federal Government 

is viewed as that of a partner with the States 

and local agencies in supporting clearly defined 

purposes considered to be of great importance to 

the nation as a whole, such as education for na- 
il 

tional defense, improving vocational education, 
equalization of access to higher education, and 
the education of the disadvantaged. 

The strong emphasis put on the categorical 
nature of the Federal aid that it should "sup- 
plement and not supplant" explains the meticu- 
lous measures taken in order to ascertain that 
the States and localities did not use Federal 
funds as a source of general support of school 
education. This in turn accounts for the enorm- 
ous amount of bureaucratization that has gone 
into the system and the resistance on its part 
to undergo changes, for instance, switch from 
economic to achievement criteria in allocating 
funds. 



Objectives of Compensatory Education , 



>^*Tf^Stl 

trough eV< 



;tructing the NIE to "undertake a 
thorough evaluation and study of compensatory 



-440- 

y& • S5 



education programs" it also called for "an ex- 
amination of the fundamental purposes of such 
programs" (NIE, 1977 f, p. 42). The Institute 
wisely ducked the responsibility to investigate 
the "fundamental purposes" which, to be sure, 
are politically highly sensitive. The central 
issue here is simply how tenable the rationale 
behind the legislation is. Both the OECD team 
in general, being charged to conduct a policy 
review, and the present author are more protect- 
ed when questioning certain assumptions behind 
the compensatory education legislation. 

In his State of the Union message to the 
Congress in 196 5 President Johnson had a section 
on aid to low- income school districts which he 
suggested she Id be given high priority. He 
pointed out that lack of formal education means 
"lower wages, fl^equent unemployment, and a home 
in an urban and rural slum." He added: "Pover- 
ty has many roots, but the taproot is ignor- 
ance." Francis Keppel, at that time U.S. Com- 
missioner of Education, shared the conviction 
about the role formal education could play in 
reducing poverty and increasing life-chances of 
the offspring of the poor. He turned against 
the "myth" that "slum children were somehow of a 
lower social order of capacity than children in" 
other sectors of our economy.". Labels, such as 
"culturally deprived" and "socially disadvan- 
taged" easily become "alibis for failure to find 
effective ways to educate these children." 
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XKeppel, 1966, p. 39). 

i 

Speaking of labels, the terra "disadvan- 
taged" appears since the mid-60's to be the one 
most frequently used in referring to children 
served by programs providing in the words of the 
ESEA "financial assistance to meet the special 
needs of children of low- income • families and the" 
impact that concentrations of low-income fam- 
ilies have in the ability of local educational 
agencies to support adequate educational pro- 
grams." (H. R. Report No. 95-1753, p. 11-12). 
However, one finds that the terms "underpriv- 
iledged" and "deprived" are often used more or 
less synonymously with "disadvantaged." A typo- 
logy suggested by Martin Trow (1978) appears to 
be useful in making a proper distinction between 
"disadvantaged" and "deprived." Trow classifies 
children according to two main" dimensions: (1) 
material and financial resources of the family 
needed to enable the children to develop their 
potential, and (2) adequacy of socialization and 
education effecting the children at home and in 
school. The two dimensions generate a typology, 
where one could distinguish between fqjir groups: 
(1) advantaged children with both adequate ma- 
terial background and socialization, (2) dis- 
advantaged children with adequate socialization 
but inadequate material resources, (3) alienated 
children with adequate material resources but 
inadequate socialization, and (4) deprived, 
children with both inadequate financial re- 
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sources and socialization. The important dis- 
tinction between the second and the fourth group 
has not always been clearly made, sometimes with 
serious consequences for programs of social in- 
tervention. Such a distinction is of particular 
importance in dealing with young people during 
the later years in high school and in transition 
to working life. The reason some of them cannot 
hold jobs after completion of schooling is not 
just lack of cognitive skills, which was the 
main belief expressed when major compensatory 
education programs for teenagers were drawn up. 
Deficient socialization bringing about lack of 
motivation and reliability is of equal import- 
ance . ^ 

.1 

Briefly, the rationale behind the compen- 
satory education programs was — and. among many 
legislators still is — the following. Failure 
and/or underachievement in school is mainly due 
to poverty. The poor tend to aggregate in cer- 
tain areas, "slums" or "poverty pockets", par- 
ticularly in urban ghettos. In order to al- 
leviate their plight and provide them with equal 
opportunity to achieve on par with their materi- 
• ally more priviledged schoolmates they should 
be given extra assistance. The desired outcomes 
of such programs of special aid should be more 
equal life chances among children from all walks 
of life in American society. 
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Critique of the Rationale Behind Compensatory 
Education. 
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The logic behind compensatory education 
could be summarized in a simple syllogism pre- 
sented in the OECD Report (OECD, 1980): 

1. Poverty and school achievements are 
closely linked. 

2. Social, mobility and life chances are 
closely linked to educational attain- 
ments. 

3. Ergo : Concentrated efforts by way of 
increased' school resources, purchased^ 
by increased financial resources tar- 
geted on disadvantaged students, will 
"break the poverty cycle." 

♦ 

When President Johnson declared his War on 
Poverty he is quoted to have said: n We are go- 
ing to eliminate poverty by education. ... people 
are going to learn their way out o<f poverty." 

The two basic propositions on a close re- 
lationship between poverty and educational fail- 
ure as well as between educational attainment 
and life chances can on the basis of empirical 
evidence be questioned. Doubts raised about the 
propositions, of course, challenge the validity 
of the conclusion. 

+ * 
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Surveys, among them the Title I programs, 
indicate that material resources in the home, as 
measured by family income, are weakly correlated 
with educational achievement* at the level of a 
0.3 correlation coefficient (NIE, 19 77e, cf. 
Husen, 1975). This means that the overwhelming 
portion of variations in school achievements is 
accounted for by non-economic factors. The con- 
siderable spread at any given income level of 
school achievements was one reason why Congress 
instructed NIE to investigate the possibilities 
of using achievements instead of poverty meas- 
ures as the basis for allocating compensatory 
' funds. 

« 

More important, in the light of recent 

4 

studies, not least the ones conducted by Jencks 
and his associates (Jencks et al., 1972; Jencks 
et al . , 1979) one could further question .the 
capability of the school as an institution to 
make up for the handicaps caused by material 
and/or cultural poverty at home. This is a per- 
vasive issue in public policies which aims by 
means of social intervention to improve the life 
chances of young people.' At issue is: to What 
extent can compensatory education , consisting 
mainly of additional teaching, targeted on eco- 
nomically handicapped children, serve as a sub- 
stitute for reforms that would affect the social 
and economic order at large? After all, one 
does not need to be a Marxist to realize that 
education does not operate in a socio-economic 
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vacuum! 

Another overriding philosophical issue, to 
which the NIE Reports did not address them-' 
selves, but which is germane to the * fundamental 
purposes" of the programs, is equality in edu- 
cation and in life chances,- what this means, and 
how it caiv.be achieved. What kind of equality 
is intended: equality of opportunity or equal- 
ity of results? To what extent is formal* equal- 
ity in educational opportunities conducive to 
equality in life chances? Provisions of formal 
equality of opportunity in education does by no 
means guarantee equal results. Some children 
are, to use an Orwell ian travesty, from the out- 
set "more equal than others," for instance, by 
being born by parents possessing privileged 
status, not to -speak of possessing more favour- 
able genes. 

The traditional confidence in what educa- 
ti&n can do has led Americans to the belief that 
by removing educational disadvantage one would,, 
bring about more social equality. At the level 
of sophistication where legislation was con- 
ceived, equality was of the classical, liberal 
brand. Equality of opportunity is traditionally 
conceived in terms of conditions that prevail in 
the student 1 s background: his social class, 
ethnic group, sex, etc. None of these shduld be 
allowed to be an advantage or disadvantage in 
shaping a person 1 s educational and vocational 
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car.eer. Merit, that is to say, proven ability 
not background, should count in getting access 
to education and to subsequent jobs. Thus, pub- 
lie policy in education should try to remove the 
economic and other barriers that prevent those 
who grow up under disadvantaged conditions to 
obtain in the first place the formal eduction 
that matches their innate potential. The di- 
lemma, which one has not until recently begun to 
recognize more clearly, is that all criteria of 
merit — test scores, achievements in examina- > 
tions, and school marks — are significantly 
correlated with sScial background. This means 
that whatever selection or self-selection that 
„ takes place for entry to an educational insti- 
tution -as well as' achievements in that institu-> 
tion is correlated with home background. 

« 

The legislative programs on compensatory 
education launched in tlte mid-19 60»s aimed at 
bringing about more equality of results,* in 
-terms both of educational and vocational at- 
tainments. This should be achieved by providing 
extra funds for children in poverty areas who 
were lagging behind in their school results. 
What actually could be achieved by providing 
extra funds was limited by two conditions. In 
the first place, children meeting the poverty 
definitions are to a large extent found in lo-. 
calities where " aTl students in public schools 
are disadvantaged with regard to initial finan- 
- cial support. Extra funds to poverty-ridden 
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districts and schools therefore do not make up 
for the initial inequality. But there is^lso a 
fundamental limiting condition. In the debate 
stimulated by OECD programs ' as viell as through 
recent empirical studies (Jencks et al. , 1972^$f 
Sewell e£ al. , 1976; Jencks et al . f 1979) it has 
been shown that equality of opportunity brought 
about <by formal equality in access and reppval^ 
of economic barriers is no guarantee that other 
forms of equality will folTow. Programs de- 
signed to ameliorate, or* even eliminate, obsta- 
cles against access to further education are not 
enough in overcoming inequalities. Again, some 
are from the outsets not least by family back- 
ground, "more equal than others." 



Compensatory Education Programs: A Brief Oyr- 
view. 

I shall not he^e try to give a detailed de- 
scription of the Federal programs for the edu- 
cation of poor and disadvantaged students in 
America. The National Report (OE, 1979) pre- 
pared for the OECD review identified some 50 
programs serving the educationally disadvantaged 
and in six excellent* ca^e studies dealt with ten 
of them in detail. The NIE evaluation studies 
are, as pointed out above, almost entirely lim- 
ited to the Title I program which ib absorbing 
some 80 percent of funds going into, compensatory 
education and therefore deserves to be treated 
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a'j if it were the only major program. It 
should^, however, be pointed out that the Head 
- Start program is still operating, with, funds 
that are, if" not absoluteJy. so, relatively heav- 
ily reduced, in comparison with the resources ten V 
years ago. ^ Among the compensatory programs ilr 
elementary and secondary school the Title VII 
program,* bi-lingual education, is of pivotal im- 
portance, for Hispanics,^ who tend to become the 
biggest minority in America. The National Re-' 
port deals vifcth two . programs in higher educa- 
tion," namely the Basic Educational WportruniJty 
Grants program and the Special ^Servicejs for 
Disadvantaged Students. Given the scope *of my 
assignment to review the NIE reports on corapen- * 
satory education, " which have by and Jarge been 
limited to Title I, I shall" also limit myself tcr\ 
the Title I program, originally launched within 
* the framework of the ESEA. 

J . - v . •• 

The by- tar largest compensatory education 
effort under Federal auspices' has been th,e Title 
I pf the ESEA passed by Congress, in 1965. ( Title 
T provided funds for special educational efforts 
V^i-ft "^sc^iools .located in arias . with high 
\^ * concentration bf poor students according to four 
criteria: (1) number of children*' aged ,5 to 17 
' in a givln county who are below the difshansky 
poverty line 2 >, tvo- thirds of the* children in 
the same age bracket who receive* payments lender 
Aid to Families witl^epeoden-t^ Children (AFDC) , 
(3) children in the samp c age bracket who live in * 

4 / • 



. . 

special institutions, and («♦) the average level 
of school expenditux'e in the State. 

Theamdunt of funds going into Title I pro- 
grams hasT'been substantial , not to say stagger- 
ing." During the 1968-7 8 period Congress allo- 
cated 'rB.-U billion "dollars. • r fcy*' .1975 Title I 
funding comprised- 8 «* percent of the total ESEA 
-funding. By late -1970's the funding level had 
risen to 2.5 billion/ from "i. 2 billion ten years 

♦ -f 

earlier. Tittle* 1 alone accounts for about 3-U 
percent of the total national expenditures for 
public islementary and secondary education. 

The NfrE study on the' efficiency /of Title 1 
found that by 1976^90 percent of the school dis- 
tricts deceived Title I funds (NIE, 1978). 
. Studies" under way, conducted for the Office of 
Education, show that slightly more than 90 per- 
cent of the schools with high concentration of 
students with.mjnorrfcy background receive some 
kind of compensatory aid, either Federal or 
State funds. x Of the Nation's elementary schools 

6 8 percent obtained Title I ai^. ' An additional 

. * v 

Ik percent received other funds, mostly State 

funds with the same purpose. Of eligible child- 
ren in 'eligible schools 67 percent were reached. 
By and^large, the Title I funds to a remarkable 
extent were targeted on schools with a high con- 
centration of poor students. But the fact that 
' *a considerable number of educationally disad- 
vantaged were not beneficiaries, as well as the 
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fact that quite a few who were well off were 
serviced, raises questions about the adequacy of 
the allocation criteria, a matter to which I 
shall return later. 

° ' Title VII of ESEA, the Bi-lingual Education 
progranf, deserves special mention. It was 
launched in 1967 at the initiative of legisla- 
tors from California and Texas with the aim of 
improving education of children from non-English 
speaking homes, mostly Hispanics. The provi- 
sions were designed to support schools in pro- 
viding bi-lingual instruction. Although in 
tfheory funds were targeted for children with 
limited proficiency in English, the Bi-lingual 
^Education Act has popularly been identified as a 
i "Mexican- American project because of its emphasis 
on teaching children of Mexican (and Puerto Ric- 
an) background. 

During our s-"te visits, particularly in New 
Mexico, it was apparent that the term "disad- 
vantaged" and even more the term "compensatory" 
were resented in connection with bi-lingual edu- 
cation. A child from a Spanish-speaking home in 
New Mexico who comes from a family that for cen- 
turies has lived there and who accordingly has a 
strong cultural identity is not necessarily 
"disadvantaged" in terms of lacking basic mater- 
ial and cultural assets. Nor can one say that 
it has to be "compensated" for not having Eng- 
lish as a mother tongue. 
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*There are among those who operate the bi- 
lingual programs, and to some extent among those 
who are beneficiaries, two opposing philosophies 
about the main objectives the program. On 
the one side, there are ti who argue that 
children from non-English speaking homes should 
become proficient in English is rapidly and ef- 
ficiently as possible even at the price of los- 
ing contact with their home language and what 
that implies of alienation from their cultural 
-heritage^' On the other hand, there are those 
who rgard bi-lingual education as a means of 
maintaining their cultural identity. A similar 
controversy has emerged in Western Europe where 
many millions of children of the "guest-workers" 
have forced the educational authorities to take 
step j in providing adequate education for them 
(Rist, 1978). 

Considering the fact that the Hispanics 
soon will constitute the largest minority in the 
United States it would have been in order if the 
Bilingual Education Act had been included in the 
efforts to evaluate compensatory education. 

While the OECD review team in the fall of 
19 78 conducted its site visits, the 9 5th Con- 
gress was in its final phase of discussing the 
amendments to the various titles in the ESEA, 
which was up for a 5-year reauthorization 
(Education Amendments, 1978). In the first 
place, the Amendments aimed at simplifying the 
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paper work which, as was repeatedly noticed dur- 
ing the site visits, was sometimes excessive, 
detailed, and burdensome to the local authori- 
ties. Instead of submitting applications an- 
nually it can now be done triannually. The 
Amendments of 1978 aim at better targeting than 
before on areas with high concentration of needy 
children. Thus, the basic Federal grants to the 
States are supplemented over and above the 1979 
appropriations by extra funds in relation to the 
number of families in the State with an income 
below the National median according to the 197 5 
survey conducted by the Bureau of the Census. 
Furthermore, the basic allocation formula will, 
beginning in 1980, take 100 per cent of the 
'children from families receiving AFDC instead of 
only 2/3 which has been the provision since 
196 5. Special grants are made available to 
school districts with a high concentration of 
children from low-income families. Thus, 
counties where t%e number of such children ex- 
ceeds 20 per cent of the total are eligible for 
such grants. 

Federal legislation on compensatory educa- 
tion has right from the outset been very strict 
on two principles: Federal funds should "supple- 
ment not supplant" , and the States and lpcgl 
educational authorities shou? d be able to demon- 
strate "maintenance of effort", that is to say, 
not slacken their efforts in terms of general 
funding of elementary and secondary education. 
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A meticulous control of the" pursuance of these 
principles has resulted in excessive red tape. 
Exclusion from the first principle can now occur 
under the new provisions, in case State funds 
are used for compensatory education programs 
similar to Title I. The waiver provision for 
maintenance of effort has been modified to allow 
waiver from one fiscal year in case of "excep- 
tional and unforeseen" circumstances. Thus, a 
disruption from one fiscal year to another of a 
useful program, simply because a district or a\ 
school happened to lose a fraction of a percent 
of eligible children and thereby falls below the 
level required to receive funds, is avoided. 

Parental participation, a cornerstone in 
the original ESEA legislation, did not always 
work out according to anticipation. This might 
have been the reason why it was included in the 
NIE Compensatory Education Study (NIE, 1977f) 
which in turn was based on the 1975-76 NIE 
National Survey of Compensatory Education, which 
in its turn was based on efforts by the Stanford 
Research Institute (1971), Vanecko et al . (1977) 
of Abt Associates, and Goettel and Kaplan (19 77) 
of the Syracuse Research Corporation. Parental 
Advisory Councils (PACs) have had considerable 
difficulties in finding their identity and have 
in many places become mere formalities. In 
other places they have become the focus of 
political patronage and struggle. Therefore, 
the regulations in the 19 78 Education Amendments 
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require certain procedures for selection of 
parents on the PACs , a minimum number of meet- 
ings per year, and a minimum number of indivi- 
duals to serve on the PACs. 

" Sometimes there has been r an overzealous 
formalism with regard to which children in a 
given school or class may be served. There has 
been a tendency to categorize certain teachers 
as Title I teachers versus the "regular" ones. 
In order to reduce such tendencies, all children 
in a school where 75 per cent of them are 
eligible for compensatory education can be 
served. In a similar vein, teachers who are 
employed mainly for compensatory programs are 
required to spend a certain amount of their time 
on other activities which belong to their teach- 
ing role, such as consultations with other 
teachers and with parents and students. 

Targeting of Compensatory Education . 

The difficulties in implementing Federal 
commitments to the education of the disadvan- 
taged can be fully appreciated only if one 
realizes the enormous complexities involved all 
the way from policy making, preparation of reg- 
ulations, assessing needs for support, distribu- 
tion of funds, planning programs at the local 
level, and evaluating and accounting upwards. 
Federal aid in education on a massive scale has 
a short history. The planning of legislation 
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and the launching 'of -'the ensuing programs 
occurred within a short time span in the early 
1960's. As always, once certain practices have 
been established and entrenched, it is difficult 
to change them. Any more deepgoirg change in 
the administrative machinery is felt as an 
intrusion upon established empires and spheres 
of influence. In addition, politicians take 
particular interest in programs that in a tan- 
gible way affect their particular constituen- 
cies. The OECD team could not avoid noticing 
how keenly aware congressmen were about the 
effects that proposed changes in the allocation 
criteria and regulations could have on their 
constituencies. This explains the reluctance to 
change existing practices of distributing funds. 

Compensatory education funds undery^ Title I 
of ESEA are distributed according "to an alloca- 
tion . formula Congress agreed upon in the 1960's 
and which, with minor modifications, is still 
employed. The poverty statistics are obtained 
from census data that could be more than ten 
years old . This means that in certain areas the 
composition of the population might have changed 
, rather drastically in a' society as mobile as the 
American. Funds are * first allocated to' the 
State, which in its turn allocates them to the 
various school districts which then allocate 
them to the various school sites. 

The number of poor, as assessed by census 
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data at the county level, is the first criterion 
of identification. Schools within districts are 
ranked according to incidence of poverty. 
Available funds are # then distributed according 
to this rank order. This means that a consider- 
able nuniber of disadvantaged children, beyond 
those who live in areas or districts not eligi- 
ble according to the poverty criterion, are not' 
serviced. Thus, there are two categories of 
educationally disadvantaged children who are not 
covered by Tit la I provisions: (1) children in 
schools not eligible or not covered by the pro- 
gram, and (2) children in eligible schools for 
whom funds r are insufficient. The NIE (1977c) 
'study that addresses itself to this problem 
found that about one-third of the eligible 
children were not served by Title I programs. 

Let me take one example from the site 
visits paid by the OECD team. When interviewing 
people at the Board of Education of the Los 
Angeles Unified School District we were told 
that by 1978 in L.A. only 172 out of more than 
• UOO elementary schools and some 30 per cent of 
the eligible children were served by" State pro- 
grams. The criterion for distributing State aid 
to schools for compensatory education, which in 
California* at least until then was comparatively 
high, was in L.A. a composite of (1) AFDC, (2) 
number of free school lunches, (3) property 
assessment, and (4) family income. After the' 
eligible schools had been identified by poverty 
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criteria, students to be served were' determined 
by the grade 6 reading scores over the last »+ 
years. 

« 

A survey on a national, representative 
sample conducted by the Systems Development 
Corporation (SDG), still under way when the OECD 
team visited Los Angeles, found that two- thirds 
of all elementary schools obtained Title I 
funds. ^ * 

There is evidently a. trade-off between 
horizontal coverage and concentration, which has 
a bearing on the practice of letting all 
children who score below the 50th percentile in 
eligible schools become eligible for services. 
We were informed that a study subcontracted to 
SDC shows that 50 percent of the children who 
are poor and score below the 50th percentile are 
serviced as compared to 35 percent who accord- 
ing to the formula are not poor and score in the 
same range. This gives reason to question the 
adequacy of poverty measures for identification 
of needy children. , 

Another aspect of mistargeting related to 
level of reading* competence is the following. I 
learned at the interview session with SDC that 
the survey had shown that 12 per cent of the 
poor~^*who score above the 50 percentile were 
serviced. The cutoff point was set at the 50th 
percentile, which seemed to be remarkably high 
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and was brought in question at the "confronta- 
tion meeting" in Paris. The 50th percentile as 
cutoff score is not consistent with The defini- 
tion that the Office of Education employs in its 
19 77 Annual Evaluation Report to Congress: "The 
term (disadvantaged) applies to children and 
adults who typically cannot succeed in the 
traditional education systems and programs." 
( op, cit . , p. U). It would seem mqre in line 
with the intentions <$Y the legislation to^place* 
the 'cutoff point for educational failures some- 
where between the 15th and the 2 5th percentile. 
Such a practice would also be more compatible 
with the 'standards set in the minimum competency 
testing programs. 

Unfortunately, the NIE evaluation studies 
have not addressed themselves to the problem of 
horizontal coverage and to what could be gained 
in services for the most needy with more con- 
centration on them by a lowe^ cutoff score. 
Again, the resistance from many school dis- 
tricts, . who at least get some "sprinkling" of 
the funds, might have been very strong with 
repercussions on the respective congressmen. 

The OECD group noted a striking lack of 
continuity in the vertical coverage of Title I 
programs and therefore in sustained efforts 
throughout the eligrole~students • entire school 
career. Compensatory education funds are con- 
centrated at two completf ly separated stages of 
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the educational career. Title I funds are 
about 80 percent used in grades K through 3. In 
spite of the fact that there are„no legislative 
barriers against using it at later stages, the 
funds under Title I provisions are, almost 
entirely spent in the elementary school through 
grade 6. Very little is spent on compensatory 
education measures at tl?e secondary level , where 
poor students from deprived background represent 
a serious problem. As long as students are in 
elementary school, attention is paid to the de- 
velopment of the 4 r basic skills in, for 
instance, reading. This objective seems to get 
out of focus at the junior high school level, 
and many students enter senior high school with 
very low reading competence, a fact which in 
many States has inspired recent legislation on 
minimum competency testing. 

I have earlier pointed out the trade-off 
between horizontal coverage and concentration in 
distributing compensatory education funds. 
There is also a trade-off related to vertical 
coverage: the number of students that can be 
covered , versus continuity and sustention of 
services. Looking back at the experience . 
education Ralph Tyler pointed out that "im- 
provements in learning can be maintained and 
"improvements in learning can be maintained and 
^ increased when the program provides for a 
sequence of three to four years rather than 
one-shot efforts to help." (Tyler, 1974, p. 
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170). He estimated- that bririgi/ig learning gains 
of the disadvantaged up to the national average 
would require a 50 percent higher per pupil 
expenditure than for the average student. 
Statements on the effect of increased sustention* 
of efforts have been made by experts on early 
childhood education with particular* reference to 
Head Start programs (Bronfenbrenner, 197U). jit 
would seem that sustention could' be 'bought at 
the price of reducing horizontal coverage by 
lowering the, cutoff score. 

V 

During the 1970 f s five pational commissions 
in the United States have dealt with problems 
of secondary education (Coleman' et al. , 197>+; 
Brown et al . , 1973; Martin et; al . , 197U; NASSP, 
1972; Weinstock, ed. , 1973). One official in 
HEW referred to the junior high school as the 
"disaster area in American education." It is the 
stage when neglect of basic skills begins to be 
strongly felt. At the end of this stage 
dropouts take place. But even before, the 
increased absenteeism means that dropout has de 
facto started. 

Problems of dropout and unemployment among 
young people in the age range 16 through 21 are 
issues with serious 1 implications for American 
education, as has been evidenced by the com- 
mittee set up by the N^ional Academy of Educa- 
tion and chaired by Clark Kerr (NAE, 1979). 
/Bridges between the formal educational system 



and the world of work' have to be built in order 
to bring young people, who have grown up in a 
weak and unsupportive environment and who with 
Trow * s typology are ^really deprived, into pro- 
ductive work. One notes that the Youth Employ- 
ment Demonstration Act of 1977 conducts experi- 
ments in five cities witft programs where young 
people are guaranteed either work or a place in 
school. It should also be mentioned that under 
the provisions of the Youth Act considerable 
funds are available for innovative projects 
linking school and woVk. Establishment of voca-' 
tional-technic'al centers and the quest for 
building life-role competencies" in senior sec- 
ondary school are other examples of efforts in 
the same direction. '"\ 

Jo achieve such objectives -as better ver- 
tical coverage by sustained efforts *awi includ- 
ing upper elementary school as well as secondary 
school, a program integration which goes far 
beyond the confines of both the Off ice of Educa-^ 
tion and HEW has to be achieved* The establish- 
ment of a Department of Education has in this 
particular respect been a step in the right 
direction. But compensaVory programs at' the 
secondary level would require a better coordina-| 
tion between programs which until now have been 
operated in separation by HEW. and the Department 
of Labor, respectively. 

' Iri the instructional Dimensions Study NIE 
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*(1977d) reports a 7-month follow-up of students 
in 'grades 1 and 3 in 1«* school districts^ Cer- 
J tain practices,' such as "pullout instruction" as 
compared to "mainstream instruction.*' were 
studied. Since, as was noticed, .by the OECD 
team, pullout is overwhelmingly used,' the re- 
sults of such f' comparison have central strate- 
gic importance! The study was also designed to 
'assess the increments in student competence from 
•the beginning of the fall through the'end of the 
.spring semester. % 

The study came 'out with two main findings. 
In the first place, it was found that compensa- 
tory education students, in the wording of the 
NIE Final Report, "make significant achievement 
' gains over a school year period" (NIE, 1978, p. 
79). These effects were larger than those 
obtained in previous studies' of Title I students 
In interpreting this finding .one should, how- 
ever, keep two important 'conditions for fhe 
study in mind. Although the Sample contained 
students from 00 classrooms in urban and rural 
districts.it was by.no means nationally repre- 
sentative and did ,not purport to be so. The 
Fina#Report points out. that the "programs ex- 
amined were purposely selected for certain in- 
structional features an<5 cannot be considered a 
representative sample of. Title I reading and 
mathematics services."' (NIE,-* 1978, p. 80). 
Secondly, and" probably more important, 10 per- 
cent of fhe* pompetisatory education students, 
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whose initial test scores were above average, 
were excluded from the follow-up. As justifica- 
tion for doing, so it is pointed out that the 
rates of gain among the excluded students were 
"similar" to the ones among the rest. If com- 
pensatory education should make sense, those who 
scor.' below average ought to show considr.r- 

g, . . ! . 'ing subjected to special- treat- 

ment. We >re now left without evidence about 
he effect that this special treatment leads to. 
ove: and above those being achieved by regular 
instruction. 



Evaluation of Compensat r^ y Education Programs . 



Needless to point out, evaluation exercises 
are by their very nature beset with value pro- 
blems as well as with technical problems. It is 
difficult enough to identify and operationa'lize 
criteria supposed to measure the objectives one 
sets out to achieve in social -intervention pro- 
grams. But an even more troublesome problem, 
although of another kind,^is encourtered in the 
attempts to identify the objectives and the 
values embedded >n these criteria. They are 
usually anchored in the middle of political con- 
troversy. Let me take one example from Sweden 
that relates to the legislation in the 19 50 f s on 
a common, comprehensive school in Sweden (Husen, 
1962). When an education bill proposing a 
change-over from a dual or parallel school sys- 
tem of basic education to a common, comprehen- 
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sive 9-year school was discussed by the Swedish 
Parliament in 1950, it met, as could be ex- 
pected, resistance on the part of the conserva- 
tives. A Special Education Committee reviewed 
the Bill and presented its review to the Parlia- 
ment, where unanimity was bought at the price of 
clarity. The first recommendation made by the 
Committee had the following .wording: "Actions 
should be taken within a certain time, which is 
to be decideci upon by the Parliament to in- 
troduce a comprehensive school based on nine 
years of mandatory school attendance. This 
school should, according to the extent the 
planned experiments prove its suitability, re- 
place elementary school, the continuation 
school", etc. In order to achieve consensus the 
Committee Chairman had inserted the words "ac- 
cording to the extent...". The proponents of 
structural change thought that a decision had 
been taken to introduce the comprehensive school 
and that certain modifications could be en- 
visaged depending upon the outcomes of the 10- 
year pilot program which was included in the 
Bill. The conservatives interpreted the recom- 
mendation to the effect that adoption of the 
very idea of the comprehensive school should 
depend on whether the pilot program proved to be 
"superior" in terms of making its students more 
"competent" than the previous types of schools, 
particularly the selective academic lower secon- 
dary school. Since a decision was taken that 
the pilot program should be accompanied by ex- 
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tensive comparisons between the new and the tra- 
ditional types of schools , the ensuing evalua- 
tion surveys during the 1950 f s were followed 
with keen interest by both opponents and pro- 
ponents * The comparisons were made in terms of 
student cognitive competence in certain key sub- 
jects, and nobody questioned the validity of 
comparisons between school types within certain 
respects quite different objectives. The Social 
Democrats wanted a school which contributed to 
equalization and conceived the comprehensive 
school reform in social terms, whereas the con- 
servatives discussed the reform in terras of in- 
structional and pedagogical efficiency. 

Similar problems were encountered in eval- 
uating the Title I programs. I have earlier 
discussed the philosophy behind them. Extra 
funds to support disadvantaged children are ex- 
pected to improve their educational attainments. 
This would in turn improve their coping power as 
adults, make them better prepared for working 
life, and lift them out of poverty. Therefore, 
the ultimate criterion of how "effective" Title 
I is would be adult occupational status and/or 
earnings of students serviced by the programs. 
This would require a longitudinal study which 
would administratively be difficult and costly 
to conduct - not to mention that it would not be 
attractive to politicians who prefer instant re- 
sults within the short perspective. The length 
of their term of office tends to determine the 



length of their perspective. 



'inerefore, practically all evaluation ef- 
forts have opted for short-range criteria, 
thereby using the one that is .close at hand: 
student achievement. In most cases one has 
tried to assess what happens to the children in 
the program over one school year. Even that can 
be a tricky exercise considering the high 
turnover of children in the categories serviced 
by Title I. When the turnover is about one hun- 
dred percent, which occurs in some schools, t v *e 
quest for "sustained efforts" indeed tends to 
become an empty phrase. 

The research literature on evaluation of 
various compensatory education programs, par- 
ticularly Head Start, Follow Through, and Title 
I, is, indeed, extensive. If one includes the 
annual reports submitted by the local educa- 
tional authorities and by the U.S. Commissioner 
of Education, the documentation is enormous. In 
addition, commissioned studies - , many of which 
were designed in grand format and at great ex- 
pense, have been conducted. It would be highly 
presumptuous to pretend to cover even the major 
relevant publications. Nor could I possibly 
pretend to come up with my own evaluation of 
studies into which impressive competence and 
considerable sums of money have been invested. 
I shall therefore mainly confine myself to gen- 
eral observations and to raising some issues. 



There have been many small-scale evalua- 
tions of Head Start programs. The first more 
extensive and significant attempt to assess the 
impact of these programs was the Westinghouse 
Report of 1969. Little and Smith (1971), who 
were commissioned by OECD to conduct a study of 
the strategies of compensation employed in the 
United States, "conclude from ^:he Westinghouse 
Report: "The overall findings of the study sug- 
gested that the summer programs had .little sus- 
tained effect; in general no differences could 
be distinguished between Head Start and non-Head 
Start children." (p. 52). Although Head Start 
children were slightly ahead of non-Head Start 
children at the beginning of grade one, by grade 
two there were no significant differences in 
school achievement. Similar results were ob- 
tained with regard to affective outcomes. As 
usual in studies of this kind, it was beset by 
methodological flaws, such as lack of pretest 
scores and one-sided emphasis on achievements. 
Tremendous variations between programs defied 
attempts to relare specific treatments to 
specific outcomes. Thus, so far the Head Start 
evaluations have not yielded any consistent 
and/or conclusive results that could serve as 
guidelines for policy makers. One should remem- 
ber, however, that in this respect attempts to 
evaluate effects of treatments similar to the 
ones that go on in the regular school setting 
have also given inconclusive results, leading to 
the desperate conclusion that "schooling does 
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A massive attempt using a longitudinal re- 
search strategy has recently been launched by 
the Education Commission of the States. Pre- 
liminary findings seem to indicate more positive 
effects of early childhood intervention than 
found in previous investigations. 

The impressive resources put at the dis- 
posal of the researchers commissioned to conduct 
evaluation studies have not been able to offset 
the shortcomings that stem from being put into 
the straight- jacket of policy makers and 
planners. The demand for quick and "relevant" 
results has tended to sterilize much of this 
rese4§ch. The evaluation of the "planned varia- 
tion" field experiments conducted under the 
auspices of the Follow Through and criticized by 
House et al. (1978) serves as a good illustra- 
tion and shall therefore be dealt with somewhat 
more in detail. 

The Follow Through program was launched as 
an attempt to sustain the effects achieved by 
Head Start in providing continuing help to for- 
mer Head Start children as they entered regular 
school. Follow Through was originally conceived 
as a program providing additional compensatory 
education in Kindergarten and in the first three 
grades of elementary school for children who had 
previously participated in Head Start. In 1968 

-469- 



it was deciced to make the program more experi- 
mental and to try out various models in early 
childhood education that should subsequently be 
evaluated (cf. Rivlin and Timpane, 1975). It 
was decided to try out about a doifetfTmodels, and 
15 million dollars were allocated to the entire 
program* By 1978 the^ annual allocation had in- 
creased to 59 million. The total, program costs 
oVer the decade since the inception amount to 
roughly 500 million dollars. Over the entire 
period 20 sponsors had worked with projects. on 
180 school sites. The Abt Associates , Inc. was 
commissioned to conduct the evaluation that 
comprised a sample of some 20 , 000 students who 
were followed up over a U-year period. The 
total evaluation costs have been estimated to be 
somewhere between 30 and 5 0 million dollars. 
The evaluation exercise reported in half a dozen 
massive volumes has been subjected to criticism 
by House crt al. (1978). Their report, sponsored 
by the Ford Foundation, as well as rebuttals 
from the researchers responsible for the Abt 
Associates evaluation have been published in a 
special issue (No. 2, 197 8) of the Harvard Edu- 
cational Review . This issue makes an extremely 
interesting reading, because it amply illus- 
trates the point above on the relationship be- 
tween educational research and educational de- 
cision-making. Researchers were in the e first 
place called upon to make their contribution ex 
post facto , when the models had been decided 
upon and with little possibility of influencing 
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either the design of the "planned experiments" 
or the way they were conducted. Very often the 
conduct of the experiments was arbitrarily ' 
changed due to particular local . circumstances 
and interests that had little to do with the 
aims of the "experiments" . 

A host of troublesome methodological prob- 
lems were encountered in the attempts to eval- 
uate conclusively the Follow Through programs. 
A few major problems and flaws deserve to be 
mentioned. In doing so I am largely drawing 
upon the report by House et al. (19 78). There 
was no systematic selection of school sites 
and/or teachers within schools. The control 
groups were grossly mismatched with the experi- 
mental groups. Adequate evaluation instruments 
were not developed from the outset and focused 
mainly on cognitive competence. On the- whole 
the outcome domains were poorly covered. The 
goals implicit in the various experiments within 
the same models were so diverse that it was very 
' difficult, not to say impossible, to develop 
common instruments that could adequately cover 
the entire range. The aggregation level repre- 
sented another matter of concern: the student 
instead of the classroom was used as the unit of 
analysis. This was inconsistent with the sta- 
tistical technique used in adjusting the means 
of the control groups by means of analysis of 
covariance. Half the original sample of chil- 
dren was lost during the 4 -year follow up. The 



hopeless situation for the evaluators was re- 
flected in the fact that the variation in mean 
achievement within models was greater than be- 
tween models. This reflects the role played by 
contextual factors for the outcomes. Such con- 
clusion is drawn by the Abt Associates, Inc. re- 
searchers (Anderson et al . , 1978) in their reply 
to the criticism leveled by House et al. (1978): 

"None of the seventeen models in the 
' evaluation demonstrated that it could 
compensate consistently for the aca- 
demic consequences of poverty... (We) 
conclude that the Follow Through 
strategy of externally sponsored cur- 
ricular change is not a reliable tool 
for raising the test scores c poor 
children. . . Local circumstances and 
behavior clearly have more to do with 
children's test performance than do 
the intentions, theories, and rhetoric 
cf outside interveners.... Taken as a 
whole, the evidence presented in our 
report (Stebbins et al. , 1977) sug- 
gests that if a local school system 
has "the potential for effective com- 
pensatory education, then outside . re- 
sources of the Follow Through kind can 
sometimes catalyze this potential. 

# 

But if not, intervention seems likely 
to be somewhat disruptive and counter-- 
productive." (Anderson et al . , 1978, 
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This statement is in line not only with 
common sense but also with findings from re- 
search on the determinants of educational inno- 
vations. The contextual factors, the ecology of 
the educative process going on in the school-, 
have to be taken into account. The involvement 
of the parents and the extent to which they in 
cooperation with the school are participating in 
the education of their children is of pivotal 
importance. 

Congress in its 19 74 Education Amendments 
requested, as mentioned earlier, the National 
Institute of Education to conduct a study of 
compensatory education, its "fundamental pur- 
poses and effectiveness." In time for consider- 
ation in connection with the reauthorization of 
the compensatory education programs a final com- 
prehensive report was submitted in September 
1978 (NIE, 1978). The NIE reports reflect a 
shift towards a widened perspective in conduct- 
ing evaluation. The NIE compensatory education 
study is in the first place not an "in-house" 
one in terms of original research endeavor with 
what that implies of research design, data col- 
lection, and statistical analyses, but rather an 
attempt to collate the extensive research con- 
ducted in some 35 projects,, most of them com- 
missioned by the Federal Government and con- 
ducted by outside organizations. 
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As has been repeatedly pointed out, most of 
the evaluation projects have mainly been pre- 
occupied with the effects of the compensatory 
education programs on student cognitive develop- 
ment as measured by achievement test scores in 
the. basic skills. This has to be seen in the 
context of the emphasis on such skills not only 
in the legislation but also in the actual opera- 
tion of Title I programs. 



The bulk of the Title I resources has gone 
into remedial reading. Recent "back-to-the- \ 
basics" battle cries have reinforced the ten- 
dency to look at the outcomes in terms of stu- 
dent performance, particularly in reading and 
mathematics. Thus, the effective and social 
development of the students, which, of course, 
is less tangible and therefore more difficult to 
measure, has tended to be neglected. Not least 
the Follow Through evaluation exercise points at 
the importance of changes in the "ecology" or 
"climate" in schools and communities where 
compensatory programs have operated. Parental 
involvement in the education of the children was 
"enhanced by the Follow Through, where parents 
. twice as often as in the control group 
. volunteered to help in the school. Many 
low-status parents became interested in improv- 
ing their own education. Another important by- 
product seems to have been that some workable 
comprehensible models of primary education were 
developed under the auspices of Follow Through. 
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The program could thereby serve as a kind of 
curriculum and instrument development labora- 
tory. A third by-product seems to have been the 
opportunity that various planned variations ex- 
periments have provided for effective on-the-^ob^ 
inservice training for teachers. 

During a visit to Systems Development Cor- ; 
poration the OECD team had the 'opportunity to 
meet with the researchers commissioned to con- 
duct a comprehensive study* of the sustained ef- 
fects of Title I basic skills. Base line data 
collected for a nationally representative sample 
of students and parents of . a considerable size 
make it possible to survey on the basis of what 
criteria services in the Title L» programs are 
delivered. This is an impressive attempt to 
overcome some of the technical shortcomings that 
have beset some previous studies. It also tries 
to map out what goes on in the home between par- 
ents and children which, regrettably, because of 
the sizo of the sample, has to be studied rather 
extensively. In our conversation with the re- 
searchers we learned that the longitudinal em- 
phasis, which was a major purpose for launching 
the study, might because of the High costs have 
to be played down.' One must regret this if it 
occurs and hope that it might be possible to 
conduct a follow-up study on ' at least a sub- 
. sample in order to assess the sustained effects 
of the compensatory education programs. 

f 
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Concluding observations . 

i 

Both in reviewing policies for compensatory 
education and in the attempts to evaluate the 
outcomes a certain amount of generosity — and 
humility — on the part of the researcher has to 
be exercised. Legislation and its implementa- 
tion in terms of regulations are outcomes of po- 
litical compromise, a process where representa- 
tives of States and districts have tried to 
imize the benefits accruing to their own con- 
stituencies. .One cannot, therefore, in the leg- 
islative wording expect the same clarity and 
consistency of definition ts in a graduate semi- 
• nar of, for instance, what is meant by "equa.li- 

« 

ty w or "disadvantage" and how these terms .should 
be. operationalized when it comes to social in- 
tervention. 

In looking back on the development of Amer- 
ican education over the last two decades one is 
deeply impressed by the considerable strides 
that have been made, not least by Federal ^Ini- 
tiatives, • which have contributed to putting % 
vital issues, particularly the education of the 
disadvantaged, on the national agenda. In spite 
of frustrated , expectations , setbacks of pro- 
grams ,i criticism of falling standards, and con- 
cerns about the young people who leave school as 
functional illiterates, the faith in education ' 
among the American people has remained re- 
• markably unaffected. As an observer from abroad, 
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one cannot avoid noticing that the Proposition 
13 movement is not dire cteH^against education as 
such but Against big government (as has also 
been evidenced by Nppinion polls). * One 
indication of the sustained, . faith is the fact 
that the 1978 Federal appropriations for educa- 
tion went up 16 percent,, which was far beyond 
what was needed to make up for inflation. 

With regard to* evaluation." I would like to 
make two observations. - Tn the first place," a 
broader perspective has to be employed in, look- 
ing at the outcomes of intervention programs 
than in some of the commissioned studies. By 
and large the employment of a wider range of 
criteria of ' evaluation is called for. Irrespec- 
tive of whether one accepts or rejects the tra- 
ditional experimental model as feasible and ade- . 
quate in assessing the outcomes of planned in- 
tervention in the educational and social welfare 
demand, the problem of keeping the relevant fac- 
tors "under control" is, indeed, formidable and 
is among other things reflecte/i in the enormous 
costs incurred in evaluating the Follow Through 
models. I agree withj^the conclusion made by the 
group of researchers commissioned -to evaluate' 
the Follow Through program in saying: "Pootv 
children still tend to perform poorly* in school 
even after the best and the brightest theorists 
— with the help of parents, local educators, 
and jrhe federal funds. .. .have done their best, to 
change the. situation." (Anderson et al. , 1978, 
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^. 163). Secondly, even taking into account the 
difficulties of conducting longitudinal studies 
ecause of the high turnover of students and of 
keeping track of people who move, ^ I think that 
wellplanned, small-scale longitudinal studies 
conducted as case studies where different models 
of compensatory education are operated, would 
prove more than largescale surveys. 

We are, in spite of all efforts by educa- 
tional research, just at the beginning of our 
attempts to assess what the school is doing to 
its students. Until now, our methods have been 
extremely crude and limited to easily measured 
cognitive performances. Our notions of what 
makes a difference in student competence have 
also been rather crude because of our failure to 
assess what, for lack of a better expression, 
could be called the "climate" of the school or 
the classroom. The study of a group of inner 
London schools conducted by Michael Rutter 
(1978) marks a turning point in this respect. 
The rather meager, not to say, discouraging re- 
sults from the evaluation exercises should not 
give rise to pessimism. In his final remarks at 
the Paris " confrontation meeting" the Chairman 
of/tHe OECD team Reter ,Karmel pointed out that 
lack methods of assessing the "chemistry 1 * or 
"climate" of the schooi. * Furthermore^j*a tend 
to overlook that increments in educational 
outcomes usually are very small. We cannot 
expect minor changes^ in input to produce 

-U78- ' 

123 



dramatic results. The compensatory education 
programs constitute after all small changes in 
the larger context of formal education. 

Finally, It should be recognized that the 
educational system cannot bear the whole burden 
of bringing about social change. Educational 
reforms cannot have an impact and cannot suc- 
cessfully be implemented if they are conceived 
as if they were operating in a social vacuum. 
In order to succeed they have to be parts of 
overarching social reforms, for which they can- 
not substitute. 
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